CL-GGD: Cross-layer Semantic Geometry Distillation for Lightweight Text Embedding Reconstruction

Abstract

Lightweight sentence embedding models are critical for large-scale retrieval and semantic matching, yet aggressive compression often disrupts the semantic geometry learned by powerful teachers. Existing distillation methods primarily rely on point-wise feature matching or task-specific supervision, which fails to preserve fine-grained relational structure across samples. In this work, we propose Cross-layer Gram Geometry Distillation, a geometry-centric framework that transfers semantic structure by aligning pairwise similarity relations across teacher and student layers. By projecting multi-layer representations into a unified hyperspherical space and matching their Gram matrices under a cross-layer weighting mechanism, the student is optimized to reconstruct the teacher’s neighborhood geometry. An additional covariance regularizer is introduced to mitigate representation collapse and anisotropy. Extensive experiments on sentence embedding benchmarks demonstrate that the proposed method consistently outperforms existing distillation baselines under strict parameter and latency constraints. Theoretical analysis further establishes that minimizing Gram discrepancy yields explicit guarantees on nearest-neighbor preservation, providing a principled foundation for retrieval-oriented lightweight embedding reconstruction.