This work proposes DLR-Lock, a method where the purveyor of the model purposely replaces each pretrained MLP in their model with a deep low-rank residual network of comparable parameter count, forcing activation memory that grows linearly with depth during backpropagation, and exploits the inference-training asymmetry...
Keitaro Sakamoto, Pierre Ablin, Federico Danieli et al.· arXiv.org· 0 citations
It is shown that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training, and cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one.
Oscar Davis, D. Roos, Alice Bizeul et al.· 0 citations
This work learns a selector that is able to produce, based on context, a per-layer cache configuration towards an overall compression budget, and achieves the Pareto frontier of accuracy against cache size, against learning-free and post-hoc baselines.
João Monteiro, Louis Béthune, A. Filippova et al.· arXiv.org· 1 citation
DynaMiCS is proposed, a dynamic mixture optimizer that casts multi-domain fine-tuning as a constrained optimization problem, and achieves stronger target-domain improvements and higher constraint satisfaction than static, dynamic, similarity-based and probing-based alternatives, without a reference model, per-example s...
Eleonora Gualdoni, Sonia Laguna, Louis Béthune et al.· arXiv.org· 2 citations· ⚡1
KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in the authors' multi-turn evaluations, and is found particularly useful for dynamic model routing.
Valérie Castin, Keitaro Sakamoto, A. Filippova et al.· 1 citation
This study investigates the crucial deployment question of whether Cartridges and other CKVs can handle off-context queries, and proposes Cartridges++, simple modifications to cartridges that retain off-context abilities at small or negligible cost.
Sonia Laguna, João Monteiro, Marco Cuturi et al.· arXiv.org· 1 citation
A penalized DRO formulation in which the adversary may choose any distribution but incurs a Wasserstein penalty for deviating from the empirical distribution is studied, showing that the adversary's problem can be reformulated as an optimization problem over transport maps that push empirical samples to adversarial one...
Alireza Abdollahpoorrostam, Ehsan Sharifian, Buse Sen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.