Transformer-based language models have become the standard in Natural Language Processing (NLP). They have surpassed human performance on specific classification tasks such as named-entity recognition, question-answer, text categorization, or generative tasks such as machine translation and summarization. However, since language models are trained with significant general-purpose texts, they may have limitations in their domain-specific knowledge. Techniques such as domain adaptation can be used to improve the models to address this issue. This study presents a systematic empirical investigation of task-adaptive continual pre-training (TAPT), introduced by Gururangan et al., for Turkish language understanding, with a particular focus on the effect of the masked-language-modeling rate. Adaptation is performed in the task-adaptive setting (TAPT), i.e., continual pre-training on the unlabeled text of the target task corpus, without requiring an external domain corpus. We achieved successful results with an average increase of 2.7%. We also addressed various issues and findings related to adaptation.
Murat Aydoğan, Savas Yildirim, Tuǧba Dalyan· IEEE Access· 0 citations
Large Language Model (LLM)-based agents automate complex tasks by integrating external tools such as web browsers, e-mail clients, file readers, and APIs, but this same integration exposes them to indirect prompt injection (IPI) attacks, in which malicious instructions hidden in tool content hijack the agent. A central but often overlooked question is how defending against such attacks affects the LLM and its own task performance and computational efficiency. In this study, we design a comprehensive testbed and a layered defense, Spotlight-Guard, that combines spotlighting-based input isolation, an LLM detection-and-quarantine pipeline, and instruction integrity based on a Hash-based Message Authentication Code (HMAC) into a single framework, and we evaluate it jointly along two axes: security and LLM performance. Experiments on locally hosted 7B-class open-weight models (Qwen-2.5-7B, Mistral-7B, and DeepSeek-Coder) use Attack Success Rate (ASR) for security and benign-task success rate together with confusion-matrix-based metrics (precision, recall, and F1) for task performance, all with bootstrap 95% confidence intervals. Across a stratified, fixed-seed benchmark of 250 adversarial and 250 benign cases per configuration, the full system reduces the ASR from 36.0% to 17.2% while preserving a 97.2% benign-task success rate and raising the detection F1 from 0.749 to 0.892, demonstrating that strong protection need not degrade the model’s task performance. A component ablation isolates each layer’s contribution, an adaptive-attack evaluation confirms a low ASR (6.7%) under attacks crafted to target the pipeline, and an analysis of computational cost (model invocations per request) quantifies the efficiency overhead, characterizing the security–performance trade-off of layered defenses on open-weight LLMs.
Doygun Demirol, Murat Aydoğan· Applied Sciences· 0 citations
