Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· 0 citations· 26 references
TL;DR
Rules-Injected Instruction Tuning (RuleGPT), a paradigm that injects IF–THEN rules extracted from training data directly into the LLM's instruction sequence, allows the LLM to focus on higher-order, context-dependent interactions.
Abstract
Large Language Models (LLMs) demonstrate strong capabilities in contextual integration and multi-step reasoning, which endow them with the potential to model heterogeneous traffic data. However, the knowledge acquired during LLM pre-training is primarily qualitative and broad, and does not provide the fine-grained, context-specific quantitative dependencies required for accurate traffic prediction. This mismatch between pre-training knowledge and task requirements can reduce the effectiveness of instruction tuning, the standard approach for adapting general-purpose LLMs to downstream tasks. To address this issue, we propose Rules-Injected Instruction Tuning (RuleGPT), a paradigm that injects IF–THEN rules extracted from training data directly into the LLM's instruction sequence. For each sample, relevant rules are retrieved and encoded as learnable rule tokens that are concatenated with the input, providing explicit guidance to the model. By encoding low-order dependencies in this way, RuleGPT allows the LLM to focus on higher-order, context-dependent interactions. Experiments on real-world traffic datasets demonstrate that RuleGPT consistently improves prediction performance over standard instruction tuning and strong forecasting baselines. The code is available at https://github.com/tigerbunny2023/RuleGPT.
DynaRule is proposed, an end-to-end framework that injects the given rules into the KV cache and turns retrieval into an internal, learnable, step-wise process, and can re-attend to the most relevant rules at each step, dynamically replacing outdated ones to support more stable multi-step reasoning.
Bohan Yu, Pengfei Cao, Chen Han et al.· 1 citation
Results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance, and show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.
Niqi Lyu, Pengtao Shi, Wei Qiu et al.· 0 citations
This survey argues that context injection strategy, rather than context capacity, is the defining research challenge for long-context LLM deployment, and proposes a three-axis analytical framework revealing that injection performance is jointly governed by selection, representation, and scheduling.
Aicha Dakir, Mohamed El Hajji, Tarek Ait Baha et al.· EPJ Web of Conferences· 0 citations
This paper presents a unified technical survey of Large Language Models (LLMs), connecting three layers of the modelling pipeline that existing surveys address in isolation: internal architecture, distributed training optimisation, and downstream adaptation. Its organising principle is the dependency between these layers—how a choice at one constrains what remains feasible at the next. The survey examines fundamental mechanisms (tokenisation, scaled dot-product attention, activation functions, and normalisation, including RMSNorm and pre- versus post-normalisation placement), then the engineering of training at scale: data, tensor, and pipeline parallelism, hybrid schemes, mixed-precision training with BF16 and FP8, ZeRO-Offload memory management, activation checkpointing, and compute-optimal scaling laws together with the conditions under which they fail. The adaptation section covers supervised and instruction fine-tuning, a comparison of parameter-efficient methods (LoRA, QLoRA, adapters, prefix and prompt tuning), Reinforcement Learning from Human Feedback with its reward-hacking failure mode, alternatives including DPO, KTO and Constitutional AI, Retrieval-Augmented Generation beyond the basic pipeline, and decoding strategies. Practical configuration guidance is given for 7B, 70B and trillion-parameter regimes. Dedicated treatments of Mixture-of-Experts architectures, long-context modelling, and hardware-aware co-design close the survey, with open challenges classified by origin and severity.
Martin Lukáč, F. Duchoň, Jakub Ivan et al.· Applied Sciences· 1 citation
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.
Indraneil Paul, F. Helm, Goran Glavas et al.· 1 citation
Large Language Models (LLMs) trained using Chain-of-Thought (CoT) supervision have achieved state-of-the-art performance on complex reasoning tasks. However, the generation of long reasoning chains introduces substantial computational overhead during inference, limiting their deployment in low-latency and resource-constrained environments. This paper proposes AdaptiReason, a novel framework that dynamically compresses intermediate reasoning steps based on task difficulty and model confidence without requiring retraining of the underlying base model. AdaptiReason employs a lightweight difficulty estimator to determine the appropriate reasoning depth for each input, followed by a learned token-pruning policy that eliminates redundant or low-information reasoning steps. Experimental evaluation on the MATH, GSM8K, and ARC-Challenge benchmarks demonstrates that AdaptiReason reduces the average number of generated tokens by 3.7× while preserving 98.2% of the baseline reasoning accuracy. Furthermore, the proposed framework is model-agnostic and can be seamlessly integrated with instruction-tuned LLMs without requiring access to model parameters, relying solely on output logits for adaptive reasoning compression. The results demonstrate that AdaptiReason significantly improves inference efficiency while maintaining high reasoning performance, making it suitable for real-time and resource-constrained LLM applications.
V. A, Mithaguru, Amrita Kundu et al.· 2026 4th International Confe...· 0 citations