Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases an...
Hao-Zhan Tang, Hao Kang, Han Cai et al.· arXiv.org· 0 citations
Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases an...
Hao-Zhan Tang, Hao Kang, Han Cai et al.· 0 citations
This paper systematically investigates two principal test-time scaling axes of LLM agents: sequential scaling through extended interaction and parallel scaling through trajectory sampling, and finds that neither scaling axis can consistently yield meaningful gains from additional test-time compute in realistic environm...
Xiao-Chuan Li, Ryan Ming, P. Setlur et al.· 23 citations· ⚡1
This work proposes Agentic Context Management (ACM), a framework that equips agents with purpose-built context editing tools for lossless context management and develops a post-training pipeline that constructs high-quality demonstrations of context management and improves model performance on both agentic search and c...
Xiao-Chuan Li, Ryan Ming, Meng Chu et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.