Maintaining high levels of daily activity and physical capability have been proposed as important constituents to promote healthy brain and cognitive aging. Studies investigating the associations between brain health and physical activity in late life have, however, mainly been based on self-reported data or measures designed for clinical populations. In the current study, we examined cross-sectional associations between physical activity, recorded by an ankle-positioned accelerometer for seven days, physical capability (grip strength, postural control, and walking speed), and neuroimaging based surrogate markers of brain health in 122 healthy older adults aged 65–88 years. We used a multimodal brain imaging approach offering complementary structural MRI based indicators of brain health: global white matter fractional anisotropy (FA) and mean diffusivity (MD) based on diffusion tensor imaging, and subcortical and global brain age based on brain morphology inferred from T1-weighted MRI data. In addition, based on the results from the main analysis, follow-up regression analysis was performed to test for association between the volume of key subcortical regions of interest (hippocampus, caudate, thalamus and cerebellum) and daily steps, and a follow-up voxelwise analysis to test for associations between walking speed and FA across the white matter Tract-Based Spatial Statistics (TBSS) skeleton. The analyses revealed a significant association between global FA and walking speed, indicating higher white matter integrity in people with higher pace. Voxelwise analysis supported widespread significant associations. We also found a significant interaction between sex and subcortical brain age on number of daily steps, indicating younger-appearing brains in more physically active women, with no significant associations among men. These results provide insight into the intricate associations between different measures of brain and physical health in old age, and corroborate established public health advice promoting physical activity.