Xiang Wang

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

GenDistill, a multi-stage pipeline for distilling a pretrained Transformer into an efficient Hybrid Kimi Delta Attention (Hybrid-KDA) student, finds that log-likelihood-based evaluation consistently underestimates the gap between teacher and student, and can reverse the ranking of design choices, so conclusions drawn f...