The volume of the lateral ventricles (LV) increases with age and their abnormal enlargement is a key feature of several neurological and psychiatric diseases. Although lateral ventricular volume is heritable, a comprehensive investigation of its genetic determinants is lacking. In this meta-analysis of genome-wide association studies of 23,533 healthy middle-aged to elderly individuals from 26 population-based cohorts, we identify 7 genetic loci associated with LV volume. These loci map to chromosomes 3q28, 7p22.3, 10p12.31, 11q23.1, 12q23.3, 16q24.2, and 22q13.1 and implicate pathways related to tau pathology, S1P signaling, and cytoskeleton organization. We also report a significant genetic overlap between the thalamus and LV volumes (ρgenetic = −0.59, p-value = 3.14 × 10−6), suggesting that these brain structures may share a common biology. These genetic associations of LV volume provide insights into brain morphology.