This paper develops a rigorous information-theoretic framework for non-repetitive sequence generation and uses it to provide theoretical underpinnings for temperature-based sampling, and extends it to the practically relevant and novel online setting, where the model’s token distributions evolve and are revealed sequentially.
This work argues that future entropy is also essential for LLM decoding, and estimates its lower and upper bounds by computing the lower and upper bounds on a hidden Markov model (HMM), which is distilled as a surrogate of the LLM.
Zhizhen Chen, Daniel Mingyi Israel, Guy Van den Broeck et al.· 0 citations
Chopthin-Consensus Power Sampling (CCPS) is introduced, demonstrating that diversity-preserving resampling and diversity-aware selection are complementary mechanisms for training-free LLM reasoning.
A one-shot (finite-blocklength) channel-coding framework based on the pairwise error probability (PEP) of a decoder with randomized tie-breaking that recovers several classical one-shot bounds, including the random-coding union bound and minimax meta-converse of Polyanskiy-Poor-Verdu, the information-spectrum bounds of...
The proposed multistage decoding framework targets the failure dynamics caused by both classical trapping sets and degenerate errors supported on symmetric stabilizers, which are among the primary limitations of iterative decoding for QLDPC codes.
M. Taghipour, Dimitris Chytas, Bane V. Vasic· 0 citations
Using a finite energy message-passing algorithm, it is demonstrated numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.
Enrico M. Malatesta, A. Passalacqua, Riccardo Zecchina· arXiv.org· 0 citations
SCHUROPT is introduced, which analytically eliminates the suffix's optimal continuous response, yielding an exact groupwise quadratic with Schur-complement curvature, and achieves the highest mean zero-shot accuracy among the evaluated backpropagation free PTQ baselines.
Gunjun Lee, Sehwan Son, Younjoo Lee et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
