This paper finds that output-distribution tendencies can be partially transmitted through unrelated synthetic data, and suggests that diversity-related behavior is shaped, at least in part, by the latent model state rather than solely by decoding-time randomness.
KITE (Knowledge-boundary Instruction Tuning via Exploration), a two-stage framework that combines failure-guided data generation with boundary-aware uncertainty curation, is proposed, showing that KITE yields more stable improvement than strong synthetic-data baselines.
The concavity of the matrix-based conditional entropy functional is proved, which makes the resulting entropy-constrained projection a convex optimization problem, and a scalable mirror-descent algorithm for its implementation is developed.
An architecture-agnostic sufficient condition is established linking behavioral similarity to inference-prompt coverage, small excess population log-loss, and similar effective target distributions---a possible training-side account rather than an empirical explanation of the observed trends.
This work presents a comprehensive evaluation of analogy diversity across ten state-of-the-art open- and closed-source LLMs, and reveals a fundamental trade-off in existing LLM diversity-enhancement methods: increasing output diversity often comes at the expense of output quality.
Yuanhao Shen, D. X. de Sousa, Caio César Sifuentes Barcelos et al.· 0 citations
Evidence is provided that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected, and that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion r...
Jiahe Fan, Si Chen, Yinghao Hou et al.· 0 citations
This work identifies a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately.
Nicolas Zucchet, Hyun Dong Lee, Scott W. Linderman· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
