Several studies have reported associations between brain iron deposits and cognitive status, and cardiovascular and neurodegenerative diseases in older individuals, but the mechanisms underlying these associations remain unclear. We explored the associations between regional brain iron deposits and different factors of cognitive ability (fluid intelligence, speed and memory) in a large sample (n = 662) of individuals with a mean age of 73 years. Brain iron deposits in the corpus striatum were extracted automatically. Iron deposits in other parts of the brain (i.e., white matter, thalamus, brainstem and cortex), brain tissue volume and white matter hyperintensities (WMH) were assessed separately and semi-automatically. Overall, 72.8 % of the sample had iron deposits. The total volume of iron deposits had a small but significant negative association with all three cognitive ability factors in later life (mean r = −0.165), but no relation to intelligence in childhood (r = 0.043, p = 0.282). Regression models showed that these iron deposit associations were still present after control for a variety of vascular health factors, and were separable from the association of WMH with cognitive ability. Iron deposits were also associated with cognition across the lifespan, indicating that they are relevant for cognitive ability only at older ages. Iron deposits might be an indicator of small vessel disease that affects the neuronal networks underlying higher cognitive functioning.