Fragile X-associated tremor/ataxia syndrome (FXTAS) is a late-onset neurodegenerative disorder typically affecting male premutation carriers with 55–200 CGG trinucleotide repeat expansions in the FMR1 gene after age 50. The aim of this study was to examine whether cerebellar and brainstem changes emerge during development or aging in late life. We retrospectively analyzed magnetic resonance imaging scans from 322 males (age 8–81 years). Volume changes in the cerebellum and brainstem were contrasted with those in the ventricles and whole brain. Compared to the controls, premutation carriers without FXTAS showed significantly accelerated volume decrease in the cerebellum and whole brain, flatter inverted U-shaped trajectory of the brainstem, and larger ventricles. Compared to both older controls and premutation carriers without FXTAS, carriers with FXTAS exhibited significant volume decrease in the cerebellum and whole brain and accelerated volume decrease in the brainstem. We therefore conclude that cerebellar and brainstem volumes were likely affected during both development and progression of neurodegeneration in premutation carriers, suggesting that interventions may need to start early in adulthood to be most effective.