Across seven Transformer backbones and six data-scarce benchmarks, iBKD consistently outperforms both locality-guidance methods and general knowledge distillation baselines, with its advantage increasing as the amount of training data decreases.
Abstract
Vision Transformers demonstrate remarkable global modeling capacity but often underperform in data-scarce regimes. Distilling convolutional inductive biases from a CNN teacher provides an effective remedy while leaving the deployed model unchanged. However, general-purpose feature distillation transfers little in this setting. In CNN-to-CNN distillation, pooling, flattening, and logit-space projections remove the spatial grid that encodes locality and translation equivariance. Unlike a convolutional student, a ViT cannot readily reconstruct this structure on its own. In this paper, we propose iBKD, a distillation framework that preserves the spatial grid throughout the entire transfer process. Its core module, the Inductive Bias Attention Module, aggregates features from all student layers onto the teacher's grid using learned weights. It then enhances structural cues through channel and deformable spatial attention and injects them via convolutional cross-attention operating directly between spatial grids rather than token sets. The module is used only during training, leaving the deployed model as an unmodified ViT with no inference overhead. Across seven Transformer backbones and six data-scarce benchmarks, iBKD consistently outperforms both locality-guidance methods and general knowledge distillation baselines, with its advantage increasing as the amount of training data decreases.
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s safety profile remains underexplored. This paper investigates Cross-Architecture Distillation, specifically examining whether the robustness properties of a Vision Transformer (ViT) Teacher can be transferred to a Convolutional Neural Network (CNN) Student (MobileNetV2). Using Centered Kernel Alignment (CKA) and Fourier Spectral Analysis, we demonstrate that robustness transfer is driven by the learning of global shape biases— preserving the teacher's functional geometry—rather than strict feature mimicry. To isolate architectural bias from model capacity confounds and validate scalability to high-resolution data, our experiments are conducted on ImageNet-1k. Results reveal that a MobileNetV2 distilled from a ViT-B/16 Teacher achieves a significantly lower mean corruption error (mCE) compared to one distilled from a massive, capacity-matched ResNet-152 (p < 0.0001, Cohen's d = 21.50), despite comparable clean accuracy. Furthermore, we show that the ViT-distilled student effectively suppresses high-frequency noise, validating its resilience against adversarial perturbations. These findings establish a pareto-optimal strategy for deploying robust, lightweight models, proving that the choice of Teacher architecture is a critical hyperparameter for safety-critical edge applications.
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations
Dataset distillation compresses a large training set into a compact synthetic set while retaining its downstream utility. Most existing methods target randomly initialized networks, whereas modern vision systems often adapt frozen pretrained encoders with lightweight modules. Distilled samples should therefore preserve the discriminative geometry of the pretrained representation space, which existing generative objectives do not explicitly consider. We propose self-supervised representation-guided generative dataset distillation (SRG), a framework that translates the SSL geometry into diffusion guidance. Specifically, SRG constructs class-wise prototypes from real-image SSL representations and performs guidance through three SSL-space objectives for prototype alignment, inter-class discrimination, and intra-class assignment. During diffusion sampling, it adopts a stage-wise guidance strategy: early denoising is anchored to the latent of the real image whose SSL representation is nearest to the assigned prototype, whereas later denoising is guided by the SSL-space objectives. This division preserves the visual realism provided by the generative prior while progressively steering samples toward representative and class-discriminative regions of the SSL representation space. SRG consistently outperforms the evaluated generative baselines across multiple datasets and IPC settings. A cross-encoder evaluation further indicates transfer across pretrained representation spaces. These results demonstrate the effectiveness of representation-guided generation for dataset distillation with pretrained SSL models.
Mingzhuo Li, Guangcheng Li, Linfeng Ye et al.· 0 citations
The "train low, deploy high" paradigm offers significant practical advantages by minimizing training overhead while enabling high-fidelity inference through increased spatial resolutions. However, Vision Transformers (ViTs) often suffer from poor zero-shot generalization to unseen resolutions compared to their convolutional counterparts. We attribute this deficiency to two fundamental phenomena: intra-patch spectral drift, where image resizing suppresses discriminative mid-to-high frequency components due to interpolation-induced low-pass filtering, and inter-patch positional awareness collapse, where the interpolation of absolute position embeddings distorts spatial priors and causes the effective receptive field to degenerate into isolated patches at larger scales. To mitigate these issues, we propose the Multi-Scale Vision Transformer (MSViT), which integrates Spectral-Constrained Convolution for adaptive frequency-weighted patch embedding, Horizontal-Vertical Separable Attention to enforce a full-span cross-shaped effective receptive field, and Reparameterized Convolutional Position Embedding to provide boundary-aware spatial bias without the need for interpolation. When trained exclusively at 224 × 224, MSViT demonstrates remarkable robustness across a broad range of test resolutions, maintaining consistent and stable accuracy as the input scales from 128 × 128 up to 640 × 640. Our work underscores that explicit modeling of spectral stability and spatial structure is essential for developing resolution-flexible vision transformers.
Kai Jiang, Peng Peng, Youzao Lian et al.· Neural Networks· 0 citations
Vision Transformers (ViT) capture global image context through self-attention but are data-hungry, typically underperforming Convolutional Neural Networks (CNNs) on the small datasets common in fine-grained tasks such as batik motif recognition. This study investigates whether a ViT, trained via knowledge distillation using the Data-efficient Image Transformer (DeiT), can overcome this limitation and compete with CNNs on a small Semarang Batik dataset. A distilled DeiT-Tiny student learns from a ResNet-50 CNN teacher and is benchmarked against two CNN references: ResNet-50 itself (a substantially larger model) and EfficientNet-B0 (a parameter-matched counterpart). In establishing this comparison, we first uncover a critical dataset integrity issue: the publicly available Semarang Batik Dataset (3,020 images) originates from only 18 unique source photographs, each augmented approximately 167 times prior to publication. This near-duplication causes severe data leakage under conventional random splitting, inflating the test accuracy of all models to a misleading 100% and rendering such evaluation meaningless. We therefore introduce a source-aware splitting strategy that enforces group-level separation between training, test partitions, and evaluate all models across three random seeds for statistical reliability. Under this corrected protocol, the distilled DeiT-Tiny attains the highest mean accuracy (95.18 ± 0.30%) and the lowest variance among the three models, matching both the larger ResNet-50 (94.87%) and the parameter-matched EfficientNet-B0 (94.68%) while using only 5.5M parameters. These results confirm knowledge distillation enables a compact Vision Transformer to compete CNNs on a limited fine-grained dataset, and underscore that verifying sample independence is a prerequisite for trustworthy evaluation on pre-augmented public datasets.
Rafi Alifa Bagja, Bedy Purnama· International Conference on...· 0 citations
Cross-modal distillation from Vision Foundation Models (VFMs) to LiDAR backbones has recently emerged as a self-supervised pretraining strategy that reduces reliance on dense point-wise annotation for 3D scene understanding. However, existing distillation pipelines typically treat the VFM as a frozen feature source and train a heterogeneous 3D backbone to match fixed image embeddings, forcing the student to bridge both the modality gap and the cross-architecture gap between dense ViT token representations and sparse 3D encoders. We propose TOLiD, a self-supervised pretraining method for LiDAR representation learning that addresses this gap by coupling a LiDAR backbone with a student Vision Transformer (ViT) initialized from a frozen VFM teacher and applying supervision over compatible patch-token representations. TOLiD converts the set of point features within each image patch frustum into a token using Frustum Pooling followed by Frustum Attention, and performs token-level distillation with visibility masking. For LiDAR-only deployment, we lift token features back to per-point representations using masked bilinear sampling to avoid patches that have limited LiDAR points. We extensively evaluate TOLiD on five heterogeneous LiDAR datasets and four cross-sensor adaptation pairs, demonstrating improved transfer with frozen backbones and lightweight heads.
Sutharsan Mahendran, Darshana Priyasad, Kaushik Roy et al.· 0 citations
UDT is proposed, a U-Net diffusion transformer that combines the representation power of DiTs with the encoding-decoding benefits of U-Nets, through data-adaptive token merging for downsampling and upsampling, while preserving the DiT token dimension.
Junno Yun, Y. Alçalar, Mehmet Akçakaya· 0 citations