Sleep is important for optimal cognitive functioning across the lifespan. Among older adults (≥55 years), self-reported short and long sleep durations have been repeatedly, albeit inconsistently, reported to elevate the risk for poor cognitive function. This meta-analytic review quantitatively summarizes the risk for poorer cognitive function among short and long sleepers in older adults. Eligible publications were searched online and manually. A total of 35 independent samples (N = 97,264) from 11 cross-sectional and seven prospective cohort studies were included. Pooled odds ratios (OR) with 95% confidence intervals (CI) were derived using random-effects models. Self-reported short and long sleep increased the odds for poor cognitive function by 1.40 (CI = 1.27–1.56) and 1.58 times (CI = 1.43–1.74), respectively. Effect sizes varied across studies and may have been moderated by both study type (cross-sectional and prospective) and cognitive domain assessed. For cross-sectional studies, extreme sleep durations were significantly associated with poorer multiple-domain performance, executive functions, verbal memory, and working memory capacity. Prospective cohort studies revealed the significant long-term impact of short and long sleep on multiple-domain performance only. These findings establish self-reported extreme sleep duration as a risk factor for cognitive aging.