Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve th...
P. Olszowiec, Paulina Tomaszewska, Maciej Zelaszczyk et al.· 3 citations
A controlled study on the example of bViT model applying one weight-tied block $L$ times, which achieves accuracy parity with ViT on 12-lead electrocardiogram (ECG) classification tasks from the PTB-XL dataset under identical training protocols.
P. Olszowiec, Michal Byra, Grzegorz Gruszczynski et al.· 0 citations
It is found that standard ViTs remain preferable when FLOPs are the primary constraint, whereas recurrent ViTs offer a better accuracy--parameter trade-off under memory constraints, whereas naive recurrence collapses to near-random performance.
Grzegorz Gruszczynski, P. Olszowiec, Michal Byra et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.