A universal hybrid architecture integrating the compact TinyBERT transformer, a cross-attention mechanism, low-rank adaptation (LoRA), and a bidirectional gated recurrent unit (Bi-GRU) is proposed, capable of delivering high accuracy, interpretability, and computational efficiency for heterogeneous textual data in multi-task and multilingual environments.
Abstract
. This paper addresses the problem of efficient processing of non-numeric textual data in the context of rapidly growing volumes of unstructured information. Particular attention is given to the limitations of large language models, including their high computational cost, inference latency, and energy consumption, which restrict their practical applicability in real-world systems. As an alternative, a universal hybrid architecture is proposed, integrating the compact TinyBERT transformer, a cross-attention mechanism, low-rank adaptation (LoRA), and a bidirectional gated recurrent unit (Bi-GRU). The aim of the study is to develop a generalized deep learning model capable of delivering high accuracy, interpretability, and computational efficiency for heterogeneous textual data in multi-task and multilingual environments. The methodology combines transformer-based encoding, inter-sequence interaction mechanisms, and parameter-efficient fine-tuning strategies. Experimental evaluation confirms the effectiveness of the proposed approach across multiple tasks, including fake news detection, sentiment analysis, toxicity classification, and topic modeling. The model achieves high performance metrics, with accuracy reaching up to 99% and F1-scores around 0.98, while maintaining significantly reduced computational requirements. It is demonstrated that LoRA mitigates catastrophic forgetting, whereas cross-attention enhances the modeling of semantic relationships between text segments.
Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges. Achieving state-of-the-art performance in natural language processing with a large pre-trained model such as BERT is expensive and time-consuming, carries a large carbon footprint, and is difficult to realize on machines with minimal computational capability. This creates a barrier to training complex models for resource-constrained languages such as Bengali. However, in a complex neural model, not all edges are equally impactful, and the contributions of some of them can be neglected. Pruning promises to reduce the memory footprint of regular networks, shorten the training time of ever-growing networks, and increase inference efficiency without sacrificing comparable performance. In this work, we introduce BnBERT-iPET, a sparse few-shot language modeling approach for Bengali, and experimentally show that a lightweight few-shot-learned language model retaining only 10% of the edges of an initial model such as BERT can perform neck and neck with much larger models on challenging tasks for a resource-constrained language such as Bengali. By learning from few shots through iterative pattern exploiting training and achieving 90% sparsity with the Lottery Ticket Hypothesis pruning technique, our pruned BnBERT-iPET model proves to be a tough competitor to state-of-the-art language models such as Bangla Electra, Indic-BERT, and XLM-RoBERTa on downstream tasks over standard benchmark datasets of the Bengali language.
S. Hossain, Md Kamrus Samad, Ananya Ghosh et al.· 0 citations
LiteLLM is introduced, a lightweight transformer architecture explicitly optimized for short-text scenarios that delivers competitive performance, fast convergence, and competitive cross-domain performance across heterogeneous short-text settings.
Hussein Ala’a Alkaabi, Fuqdan A. Al-Ibraheemi, Ali kadhim Jasim· Neural computing & applicati...· 0 citations
Experiments on diverse open-source text classification datasets show that AHR can outperform standard baselines like LoRA when learning from limited samples, despite having 200-1000x fewer trainable parameters, as the authors' AHR only modifies ~0.0001% of the model's parameters.
Tuomas P. Oikarinen, Zixiao Chen, Charlotte Siska et al.· 0 citations
H-FuseNet is proposed, a hybrid multi-representation fusion framework that combines pretrained transformer representations with deception-oriented handcrafted linguistic, stylistic, and semantic features that can improve the reliability of automated fake news detection systems.
A. Mahmood, Marek Sikora, M. Ather et al.· Machine Learning and Knowled...· 0 citations
The rapid advancement of generative AI has made high-quality deepfakes increasingly accessible, raising serious concerns regarding misinformation and digital trust. Consequently, developing reliable deepfake detection systems has become a critical task in multimedia forensics. However, existing methods often face a severe trade-off between cross-dataset generalization and computational efficiency. They heavily rely on large-scale backbones, which limits their deployment on resource-constrained edge devices. To bridge this gap, we propose HiDD-Net, a hierarchical dual-domain distillation framework. Specifically, the teacher network integrates multi-granularity spatial views (global, local, and facial) via a cross-attention mechanism to explicitly align fine-grained anomalies with global context. Simultaneously, it captures subtle upsampling spectral artifacts using a three-level discrete wavelet transform (DWT) and Spatial Rich Model (SRM) features. Subsequently, we distill these fused spatial-frequency representations into a compact student model (MobileNetV3) by enforcing feature alignment in a shared semantic space. Extensive evaluations on DeepfakeBench demonstrate that the student achieves a competitive cross-dataset average AUC of 0.771 (vs. the teacher’s 0.796) with only 6.67M parameters. Furthermore, it supports real-time CPU inference (30+ FPS) and remains highly robust to JPEG compression and Gaussian blur, offering a practical solution for mobile forensic deployment.
This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.
Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner et al.· 0 citations
