The Layer Transferability Profile: Mapping Feature Hierarchies in Pretrained Vision Models

Abstract

Transfer learning with ImageNet-pretrained backbones is ubiquitous in computer vision, yet the question of which intermediate layer best serves a downstream task is typically resolved by convention rather than controlled measurement. Despite the prevalence of this practice, no systematic comparison of layer-wise frozen representations has been conducted across the modern architectures that dominate current pipelines. We present the Layer Transferability Profile, a reproducible evaluation protocol and an accompanying metric, the Layer-Task Transferability Score (LTTS), that together quantify how transferable information is distributed across depth. Applying the profile to three architectures, ResNet-50, EfficientNet-B0, and ViT-B/16, on three recognition benchmarks (CIFAR-10, CIFAR-100, and Food101), we find that transfer performance improves monotonically with depth and that the final or near-final representation is consistently optimal (LTTS 0-1.2%). This pattern contrasts with recent findings in speech and language foundation models, where intermediate layers can yield substantial gains, suggesting that the concentration of transferability in late layers is characteristic of supervised visual pretraining for recognition tasks rather than a universal property of deep representations. Notably, ViT-B/16 achieves the strongest transfer accuracy (78.1% on CIFAR-100, 94.6% on CIFAR-10, and 68.6% on Food-101) while using 768-dimensional features, outperforming ResNet-50’s 2048-dimensional and EfficientNet-B0’s 1280-dimensional representations. This indicates that representational quality per dimension varies substantially across architectures and can be more important than raw feature size when selecting a backbone for transfer.