Mixture-of-LoRA-experts methods raise the capacity of low-rank adaptation by routing each token to a few low-rank experts. Nearly all of them tie one input-side factor to one output-side factor per expert, and nearly all of them route by scoring the token: the router picks experts without seeing what any of them would...
Priya Nair, Lukas Brenner, Maya Lindqvist et al.· 0 citations
SPARCL is proposed, a spectral partitioned analytic continual learner that decomposes the running autocorrelation into a high-energy core and a residual complement, freezes old-class classifier components in the core subspace, and updates only the residual block through recursive least squares with an optional residual...
James E. Hartley, Zeropy Surio, D. Whitmore et al.· 0 citations
This work introduces CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion and improves over fixed top-k MoE-LoRA at matched compute and matches the fixed-k=4 baseline while activating fewer experts.
Tom Saliencro, Rohan Desai, Priya Nair et al.· arXiv.org· 1 citation
Mixtures of low-rank adaptation experts increase parameter-efficient capacity by routing each input through a subset of adapters. Recent dynamic routers activate more experts when the router or prediction is uncertain. This rule silently equates uncertainty with useful additional computation: an uncertain example may c...
Tom Saliencro, Rohan Desai, Priya Nair et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.