ScaleSurfer: multi-scale anatomical segmentation and parcellation of the human brain

Abstract

Human brain magnetic resonance imaging (MRI) revolutionized our ability to non-invasively probe individual differences in neuroanatomy. These anatomical scans, in turn, also allow us to accurately localize functional MRI (fMRI) activity. However, extracting anatomical labels and structural characteristics, such as cortical surface area or thickness, is a computationally demanding task, taking on the order of hours per brain volume. This is an intrinsically multi-scale problem given that local image structure defines fine boundaries, whereas accurate assignments depend on broader anatomical context. Here, we introduce ScaleSurfer, a three-dimensional convolutional vision transformer model based on multi-scale learning. Convolution blocks capture local anatomical detail and a transformer bottleneck integrates the distributed spatial context. This approach provides rapid, whole-brain morphometric feature estimation, including volume, cortical thickness, surface area, and curvature. Importantly, ScaleSurfer accomplishes this nearly five orders of magnitude faster than current pipelines, taking 150-500 ms instead of 5 hours. We validated ScaleSurfer on multiple datasets, showing stable learning across heterogeneous MRI collections, and demonstrate feasibility by training an interpretable Alzheimer’s disease classifier that identifies reductions in primarily medial temporal lobe subregions compared to healthy controls. ScaleSurfer positions multi-scale representation learning as a practical route toward faster, anatomically faithful structural MRI processing, whose speed paves the way for nearly real-time anatomical quality control during scanning.