Slowed processing speed is considered a hallmark feature of cognitive decline in cerebral small vessel disease (SVD), however, it is unclear whether SVD’s association with slowed processing might be due to its association with overall declining general cognitive ability. We quantified the total MRI-visible SVD burden of 540 members of the Lothian Birth Cohort 1936 (age:72.6±0.7 years; 47% female). Using latent growth curve modelling, we tested associations between total SVD burden at mean age 73 and changes in general cognitive ability, processing speed, verbal memory, and visuospatial ability, measured at age 73, 76, 79 and 82. Covariates included age, sex, vascular risk, and childhood cognitive ability. In the fully-adjusted models, greater SVD burden was associated with greater declines in general cognitive ability (standardised β: -0.201; 95%CI: [-0.36, -0.04]; pFDR=0.022) and processing speed (-0.222; [-0.40, -0.04]; pFDR=0.022). SVD burden accounted for between 4 and 5% of variance in declines of general cognitive ability and processing speed. After accounting for the covariance between tests of processing speed and general cognitive ability, only SVD’s association with greater decline in general cognitive ability remained significant, prior to FDR correction (-0.222; [-0.39, -0.06]; p=0.008; pFDR=0.085). Our findings do not support the notion that SVD has a specific association with declining processing speed, independent of decline in general cognitive ability (which captures the variance shared across domains of cognitive ability). The association between SVD burden and declining general cognitive ability supports the notion of SVD as a diffuse, whole-brain disease and suggests that trials monitoring SVD-related cognitive changes should consider domain-specific changes in the context of overall, general cognitive decline.