Caring for grandchildren provides grandparents an opportunity to sustain an active lifestyle and remain socially engaged in older age. Studies have examined the association between providing care to grandchildren and grandparents’ cognitive function. However, these studies had several limitations and yielded mixed findings. Using data from the China Health and Retirement Longitudinal Study, this study examined the longitudinal association between caregiving and cognitive functioning in grandparents aged 50–75 years. Grandparents’ caregiving status and cognitive functioning were assessed in 2011, 2013, and 2015. Random-effects and fixed-effects regression models were estimated and compared. The full sample (20,000?+?person-year observations) was divided into four subgroups to examine gender and rural–urban variations. Being a part-time noncoresident caregiver was associated with higher scores on episodic memory for grandfathers but not for grandmothers. Compared with their noncaregiving counterparts, rural part-time multigenerational caregiving grandmothers had significantly better mental intactness and global cognitive functioning, while rural full-time noncoresident caregiving grandmothers had significantly better episodic memory. Caregiving status was not associated with any cognitive measure among urban grandmothers. Taken together, these findings suggest that Chinese grandparents enjoy cognitive benefits from providing casual or intensive care to grandchildren, but these benefits vary substantially by gender and rural–urban status.