Aug 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP, pp. 1-18· 2 citations
Medicine
TL;DR
The Multi-Level Optimal Transport (MultiLevelOT) is introduced, a novel approach that advances the optimal transport for universal cross-tokenizer knowledge distillation and is robust to different student and teacher models across model families, architectures, and parameter sizes.
Abstract
Knowledge distillation (KD) has become a prevalent technique for compressing large language models (LLMs). Existing KD methods are constrained by the need for identical tokenizers (i.e., vocabularies) between teacher and student models, as they assume a consistent semantic correspondence across logit dimensions, limiting their versatility in handling LLMs of different architecture families. In this paper, we introduce the Multi-Level Optimal Transport (MultiLevelOT), a novel approach that advances the optimal transport for universal cross-tokenizer knowledge distillation. Our method aligns the logit distributions of the teacher and the student at both token and sequence levels using diverse cost matrices, eliminating the need for dimensional or token-by-token correspondence. At the token level, MultiLevelOT integrates both global and local information by jointly optimizing all tokens within a sequence to enhance robustness. At the sequence level, we efficiently capture complex logit distribution structures using a flexible Sinkhorn loss with relaxed marginal constraints, allowing partial transport and absorbing unmatched tokens, thereby serving as an efficient OT-based surrogate while tolerating unaligned mass. Extensive experiments on tasks of extractive QA, generative QA, summarization, instruction- following, math reasoning, and Big-Bench-Hard demonstrate that the MultiLevelOT outperforms state-of-the-art cross-tokenizer KD methods. Our approach is robust to different student and teacher models across model families, architectures, and parameter sizes. We further validate MultiLevelOT in multimodal-input settings and introduce task-specific technical extensions for both multi-teacher text distillation and structured visual-token distillation.
Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and memory costs at inference time. Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large teacher model to a smaller student model via...
Byeonghu Na, Donghyeok Shin, Yeongmin Kim et al.· 0 citations
Large language models have shown strong reasoning capabilities, but their high inference costs make knowledge distillation an important approach for transferring such capabilities to compact models in resource-constrained scenarios. On-policy self-distillation further reduces the reliance on external large teacher mode...
A four-coefficient parameterization lambda_t = sigma(a * h_t + b * u(x) + c + d * gap_t) in which direction-aligned proxies of EOPD and ToDi appear as one-dimensional (1D) restrictions, and which adds multi-channel composition and an explicit bias as further degrees of freedom.
Su-Wan Wu, Yu-Meng Lin, Peng-Cheng Yuan et al.· 0 citations
On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grain...
Ke-Di Chen, Chen Lin, Yutao Sun et al.· 1 citation
Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Leng...
Felermino D. M. A. Ali, Millicent Ochieng, O. Ekwejunor-Etchie et al.· 0 citations
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as mo...
Franck Signe Talla, Hippolyte Pilchen, François Yvon et al.· Conference on Empirical Meth...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.