This work proposes DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation, and exploits the inference-training asymmetry...
Keitaro Sakamoto, Pierre Ablin, Federico Danieli et al.· arXiv.org· 0 citations
It is shown that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training, and cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one.
Oscar Davis, D. Roos, Alice Bizeul et al.· 0 citations
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
DynaMiCS is proposed, a dynamic mixture optimizer that casts multi-domain fine-tuning as a constrained optimization problem, and achieves stronger target-domain improvements and higher constraint satisfaction than static, dynamic, similarity-based and probing-based alternatives, without a reference model, per-example s...
Eleonora Gualdoni, Sonia Laguna, Louis Béthune et al.· arXiv.org· 2 citations· ⚡1
KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in the authors' multi-turn evaluations, and is found particularly useful for dynamic model routing.
Valérie Castin, Keitaro Sakamoto, A. Filippova et al.· 1 citation
This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.
Sonia Laguna, João Monteiro, Marco Cuturi et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.