Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases an...
Hao-Zhan Tang, Hao Kang, Han Cai et al.· arXiv.org· 0 citations
This paper introduces Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information.
Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases an...
Hao-Zhan Tang, Hao Kang, Han Cai et al.· 0 citations
Jet-Long is proposed, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length via a parameter-free analytic schedule, recovering the base model exactly at short inputs while extrapolating cleanly at long one...
Lightning Weave is introduced, a post-training framework that extracts and composes independently learned capabilities in a single student through on-policy distillation and achieves a state-of-the-art accuracy-efficiency frontier.
DC-Gen, a general framework that accelerates text-to-image diffusion models by leveraging a deeply compressed latent space, uses an efficient post-training pipeline to preserve the quality of the base model to reduce the latency of 4K image generation.