AKDF: Adaptive knowledge distillation for secure, efficient, copyright-protected model publishing
TL;DR
The Adaptive Knowledge Distillation Framework (AKDF), a unified training approach that simultaneously com-presses LLMs and embeds ownership signals for copyright assurance, is introduced, providing a practical path to-ward bandwidth-efficient and ownership-aware deployment of large-scale language models.
Abstract
The rise of large language models (LLMs) has transformed natural language processing, powering applications from creative writing to code generation. However, their vast size and proprietary nature present two major challenges, including efficient deployment on limited hardware and secure protection of intellectual property. This work introduces the Adaptive Knowledge Distillation Framework (AKDF), a unified training approach that simultaneously com-presses LLMs and embeds ownership signals for copyright assurance. AKDF employs parameter-efficient Low-Rank Adaptation (LoRA) to distill a 1-billion-parameter student model from an 8-billion-parameter teacher while integrating an output-level watermarking module directly into the distillation process. This design reduces trainable parameters and encodes verifiable ownership signatures without altering the frozen base weights. Experiments on ARC-Easy, PIQA, and WMT16 show that the student retains approximately 76% of the teacher’s performance while maintaining competitive reasoning and translation quality under substantial compression. AKDF also enables secure and communication-efficient model publishing, providing a practical path to-ward bandwidth-efficient and ownership-aware deployment of large-scale language models.