Birth weight, an indicator of fetal growth, is associated with cognitive outcomes in early life (which are predictive of cognitive ability in later life) and risk of metabolic and cardiovascular disease across the life course. Brain health in older age, indexed by MRI features, is associated with cognitive performance, but little is known about how variation in normal birth weight impacts on brain structure in later life. In a community dwelling cohort of participants in their early seventies we tested the hypothesis that birth weight is associated with the following MRI features: total brain (TB), grey matter (GM) and normal appearing white matter (NAWM) volumes; whiter matter hyperintensity (WMH) volume; a general factor of fractional anisotropy (gFA) and peak width skeletonised mean diffusivity (PSMD) across the white matter skeleton. We also investigated the associations of birth weight with cortical surface area, volume and thickness. Birth weight was positively associated with TB, GM and NAWM volumes in later life (β ≥ 0.194), and with regional cortical surface area but not gFA, PSMD, WMH volume, or cortical volume or thickness. These positive relationships appear to be explained by larger intracranial volume, rather than by age-related tissue atrophy, and are independent of body height and weight in adulthood. This suggests that larger birth weight is linked to more brain tissue reserve in older life, rather than age-related brain structural features, such as tissue atrophy or WMH volume.