Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers
TL;DR
Soft Mixture-of-Recursions (SoftMoR) is proposed, which learns token-wise weights to combine representations across recursion steps, and instantiate it as the Soft Recursive Vision Transformer (SR-ViT), which can benefit from deeper recursion while adding few parameters.
Abstract
Vision Transformers are typically scaled by increasing depth or width, at the cost of more unique parameters. We explore recursion as an alternative scaling axis that increases effective computational depth through parameter reuse. Yet simply increasing recursion depth does not reliably improve performance because representations from different recursion stages are not fully utilized. We propose Soft Mixture-of-Recursions (SoftMoR), which learns token-wise weights to combine representations across recursion steps, and instantiate it as the Soft Recursive Vision Transformer (SR-ViT). Experiments on ImageNet-1K, COCO, and ADE20K show that SR-ViT can benefit from deeper recursion while adding few parameters. On ImageNet-1K, increasing recursion depth from T=1 to T=3 improves top-1 accuracy from 79.83% to 82.45% with only 1.1M additional parameters. At matched effective depth, SR-ViT also outperforms independently parameterized counterparts with substantially fewer parameters. These results support recursive deepening as an effective, parameter-efficient scaling axis for Vision Transformers. The code is available at https://github.com/sangin-lee/sr-vit .
