Jul 2026· Applied and Computational Engineering· 0 citations
TL;DR
It is argued that the ability of long context should not only come from increasing the context window, but also from the ability of the model to locate, integrate and reason about important information in long text.
Abstract
In recent years, large language models (LLMs) have achieved significant results in natural language processing. They are applied to various tasks, including text generation, question answering, automatic summarization, code generation, and complex reasoning. With the increasingly complex real scenarios, the length of input text that models need to deal with also grows. Thus, the long-context processing ability of language models has gradually become an important factor in evaluating the practicability of LLMs. This paper gives an introduction to the long-context processing ability of large language models. It first introduces the background of large language models and the basic concept of long-context processing. It then summarizes the main technical methods of long-context modeling, such as improving positional encoding, training stage expansion, inference-stage optimization, and architecture-level innovation. Third, the paper also discusses the use of long-context ability in long-document question answering, long-text summarization, multi-document integration, code understanding and long-context evaluation tasks. Then, summarize the current main challenges and prospects of research work. This paper argues that the ability of long context should not only come from increasing the context window, but also from the ability of the model to locate, integrate and reason about important information in long text.
A comprehensive review of the evolution of NLP from traditional rule-based approaches to modern transformer models including BERT and GPT demonstrates that NLP continues to transform intelligent systems and is expected to play an increasingly significant role in the development of next-generation AI technologies.
P. Kalaiselvi· International Journal of Eme...· 0 citations
CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models, and argues that this combination offers a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.
Malvina Nissim, Danilo Croce, V. Patti et al.· Italian Journal of Computati...· 0 citations
MGAL is the first multilingual, granularity- and position-aware long-context benchmark, constructed from United Nations reports spanning 8K to 128K tokens across the six official UN languages, and finds that LLMs perform well at word-level tasks but struggle with coarser-grained ones.
Chunhan Li, Chenglin Xu, Zongyang Zhang et al.· 0 citations
Transformer-based language models have become the standard in Natural Language Processing (NLP). They have surpassed human performance on specific classification tasks such as named-entity recognition, question-answer, text categorization, or generative tasks such as machine translation and summarization. However, since language models are trained with significant general-purpose texts, they may have limitations in their domain-specific knowledge. Techniques such as domain adaptation can be used to improve the models to address this issue. This study presents a systematic empirical investigation of task-adaptive continual pre-training (TAPT), introduced by Gururangan et al., for Turkish language understanding, with a particular focus on the effect of the masked-language-modeling rate. Adaptation is performed in the task-adaptive setting (TAPT), i.e., continual pre-training on the unlabeled text of the target task corpus, without requiring an external domain corpus. We achieved successful results with an average increase of 2.7%. We also addressed various issues and findings related to adaptation.
Murat Aydoğan, Savas Yildirim, Tuǧba Dalyan· IEEE Access· 0 citations
Large language models (LLMs) are built on the classic Transformer architecture and have become a core driving force for the rapid development of modern artificial intelligence. This paper presents a systematic review of LLMs, elaborating on their fundamental working principles, mainstream open-source models, effective lightweight optimization methods, retrieval-augmented generation frameworks and key human-value-aligned technologies. Nowadays, LLMs have been widely applied in practice. Typical scenarios include intelligent text generation, professional knowledge-based question answering and automated code generation, delivering remarkable value to both industries and academia. However, their large-scale industrial application is still restricted by multiple challenges. The major issues involve content hallucination, poor model interpretability, excessive computing resource consumption, potential ethical risks and unsatisfactory multimodal integration capability. This paper also forecasts the future development directions of LLMs, such as lightweight deployment on edge devices, safety-focused human value alignment, in-depth cross-modal fusion and customized large models for vertical industries. Additionally, it collects a number of representative cases, which can offer solid references and practical guidance for relevant researchers and engineering practitioners to carry out further studies.
The proposed WILO-BiLSTM model can perform superior to the conventional approaches and its performance results in terms of METEOR, BLEU, ROUGE, and SPICE score at training data 90% is 0.28, 0.50, 0.56, and 26.93 for the SquAD dataset, respectively.
Pallavi Yevale, Nilesh Uke· Journal of Intelligent Decis...· 0 citations