Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling
When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and caus...
Tingshuo Fan, Hong-Tao Mu, Tian-Yun Zhou et al.
· 0 citations