Jul 2026· Knowledge and Information Systems· Vol 68· 0 citations· 49 references
Computer Science
TL;DR
Results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures, with the distilled FCN student reducing parameters by a factor of 38, the distilled Inception student achieving nearly the same performance as the teacher with 42% fewer parameters and the distilled ConvTran student with 2 attention heads showing the most significant improvement through distillation.
Abstract
Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing. However, high computational and memory demands of state-of-the-art architectures pose challenges for deployment in resource-limited environments. Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model while maintaining competitive performance. In this work, we investigate the effectiveness of KD for Time Series Classification (TSC) across three architectures: the classical Fully Convolutional Network (FCN), the convolutional Inception model and the transformer-based ConvTran model. We evaluate our approach on UCR Archive, the largest benchmark repository of time series datasets, by modifying architectural components such as convolutional filters, Inception modules and attention heads across the three architectures. Our results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures, with the distilled FCN student reducing parameters by a factor of 38, the distilled Inception student achieving nearly the same performance as the teacher with 42% fewer parameters and the distilled ConvTran student with 2 attention heads showing the most significant improvement through distillation. To encourage further research and reproducibility, we provide our implementation at https://github.com/MSD-IRIMAS/KD-4-TSC.
This study systematically compares seven pre-trained feature extractors across three architectural families, convolutional neural networks (CNNs), Vision Transformers (ViTs), and self-supervised models to provide practical guidance on model selection for downstream deep learning tasks.
Rafeek Sibrikhan, M. Mufassirin· Sri Lankan Journal of Techno...· 0 citations
Accurate prediction of Remaining Useful Life (RUL) is critical for predictive maintenance and minimizing downtime in industrial systems. This paper presents a cross-domain deep learning framework based on a hybrid Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN–BiLSTM) architecture. Unlike domain-specific models that require handcrafted features, the proposed framework extracts local degradation features through CNN layers and captures long-term dependencies via BiLSTM networks. The model is evaluated on three heterogeneous datasets: construction machinery, continuous casting machines, and lithium-ion batteries. Experimental results show that CNN–BiLSTM consistently outperforms baselines, achieving up to 22% lower RMSE compared to GRU and 30–50% lower RMSE compared to traditional models. On the construction dataset, it achieves an MAE of 48.2 hours and RMSE of 67.1 hours (R2 = 0.88), outperforming GRU by 20%. For the casting dataset, the model attains an MAE of 87.6 tons and RMSE of 113.9 tons (R2 = 0.87), surpassing Random Forest by over 35%. On the battery dataset, CNN–BiLSTM reduces the MAE to 49.6 cycles and RMSE to 72.8 cycles (R2 = 0.89), while also achieving the lowest Timeliness Score (27.5) and PHM08 Score (192.4). Cross-domain experiments are evaluated under two settings: zero-shot transfer, where the model is trained on one source domain and directly tested on a different target domain without using labeled target-domain samples, and fine-tuned transfer, where 20% of labeled target-domain samples are used to update only the fully connected layers while keeping the CNN and BiLSTM layers frozen. The zero-shot results reflect the effect of domain shift, while the fine-tuned results show that lightweight transfer adaptation reduces RMSE by 25–40% across domains. These findings indicate cross-domain adaptability under limited target-domain supervision rather than fully unsupervised cross-domain generalization. These results highlight the feasibility of a unified CNN–BiLSTM framework for scalable, cross-domain RUL estimation and its suitability for real-world prognostic applications.
Sudip Saha, Muhammad Arslan Pervaiz, Muhammad Safwat Rahman et al.· PLoS ONE· 0 citations
This paper proposes a time-series data processing model based on the Transformer Encoder, designed to address the limitations of traditional Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTM) in handling long sequences, such as long-term dependency issues, vanishing or exploding gradients, and low computational efficiency. By employing a self-attention mechanism along with learnable positional encoding and multi-head self-attention, the model effectively captures global dependencies in sequential data and significantly enhances its expressive power. To improve training stability, layer normalization is applied after the Transformer encoder, and the data processing flow is optimized through reorganization across batch and node dimensions. The model demonstrates clear advantages in parallel computation and long-term dependency modeling, enabling more efficient processing of complex time-series data. It exhibits superior computational efficiency and stronger generalization capabilities, particularly in real-time traffic flow prediction tasks based on large-scale datasets and long sequences. Experimental results show that the proposed model outperforms traditional RNN and LSTM models across various tasks, indicating promising potential for practical applications.
Ze Zhao, Mingyan Jiang, Feng Wang· International Conference on...· 0 citations
Deep learning has evolved into a major computational paradigm for solving complex problems involving image understanding, remote sensing, human activity recognition, graph-based learning, and other data-intensive applications. However, the increasing diversity of deep learning architectures has made model selection and performance assessment more difficult because predictive effectiveness must be evaluated alongside computational complexity, robustness, scalability, and deployment requirements. This article presents a systematic analysis of deep learning models by synthesizing existing research on fundamental deep learning techniques, neural network architectures, application-specific models, robustness, and hardware acceleration. The review comparatively examines conventional deep learning approaches, convolutional neural networks, graph convolutional neural networks, robust deep learning models, and application-oriented architectures. Particular attention is given to the relationship between model architecture and performance dimensions such as accuracy, generalization, computational efficiency, scalability, and practical deployability. The analysis indicates that no single deep learning architecture is universally optimal; rather, performance depends strongly on data characteristics, task requirements, architectural complexity, and computational resources. CNN-oriented models demonstrate strong suitability for spatial and visual information, graph-based approaches provide advantages for relational data but introduce substantial acceleration challenges, and robust learning approaches address reliability concerns that conventional accuracy-oriented evaluation may overlook. The study establishes a multidimensional framework for assessing deep learning performance and identifies the need for evaluation strategies that jointly consider predictive capability, robustness, computational cost, and deployment constraints.
D. Iliev, Dr. Elena Stojanovsk· International Journal of Nex...· 0 citations
Deploying deep learning models on resource-constrained edge devices necessitates a trade-off between computational efficiency and robustness. While Knowledge Distillation (KD) is widely used to compress large models into lightweight students, the impact of the Teacher’s architectural inductive bias on the Student’s safety profile remains underexplored. This paper investigates Cross-Architecture Distillation, specifically examining whether the robustness properties of a Vision Transformer (ViT) Teacher can be transferred to a Convolutional Neural Network (CNN) Student (MobileNetV2). Using Centered Kernel Alignment (CKA) and Fourier Spectral Analysis, we demonstrate that robustness transfer is driven by the learning of global shape biases— preserving the teacher's functional geometry—rather than strict feature mimicry. To isolate architectural bias from model capacity confounds and validate scalability to high-resolution data, our experiments are conducted on ImageNet-1k. Results reveal that a MobileNetV2 distilled from a ViT-B/16 Teacher achieves a significantly lower mean corruption error (mCE) compared to one distilled from a massive, capacity-matched ResNet-152 (p < 0.0001, Cohen's d = 21.50), despite comparable clean accuracy. Furthermore, we show that the ViT-distilled student effectively suppresses high-frequency noise, validating its resilience against adversarial perturbations. These findings establish a pareto-optimal strategy for deploying robust, lightweight models, proving that the choice of Teacher architecture is a critical hyperparameter for safety-critical edge applications.
Tamuno Opubo Dappa, Somtochukwu Anunobi· International Conference on...· 0 citations
Deep neural networks excel in various tasks but struggle to generalize across evolving data distributions, leading to significant performance degradation under domain shifts. Domain incremental learning (DIL) addresses this challenge by enabling models to continuously adapt while retaining prior knowledge. Among existing DIL approaches, the parameter-isolation paradigm achieves state-of-the-art performance. However, these methods often adopt a one-size-fits-all approach to adapt to new domains, resulting in either insufficient learning capacity or redundant parameters. In this work, we propose BPG, a unified framework that addresses both challenges through two complementary components: BPG-Adapter, which dynamically determines each domain's adapter hidden dimension based on domain-specific feature separability, and BPG-Inference, a soft domain mixture strategy that integrates multiple domain-specific models at test time, mitigating domain ID misselection. Experimental results on DomainNet, CDDB, and CORe50 demonstrate that BPG consistently outperforms uniform adapter-based approaches and hard domain selection strategies, achieving state-of-the-art average accuracy while reducing forgetting to as low as 0.22% on DomainNet.
Qiang Wang, Songlin Dong, Shaokun Wang et al.· 0 citations