Information-Theoretic Analysis of Next-Token Prediction under Markovian Data
An information-theoretic framework for generalization in next-token prediction under temporally dependent data is developed, revealing the roles of context length, model complexity, sample size, margin, and temporal mixing and yielding informative guarantees for deterministic algorithms over continuous hypothesis space...
Masoud Kavian, Abdellatif Zaidi, Milad Sefidgaran
· 0 citations