Mar 2026· arXiv.org· Vol abs/2603.12165· 0 citations· 36 references
Computer Science
TL;DR
QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query conditioned on the query, is proposed and Reverse Mutual Information (RMI) is defined to quantify the information gain about the query conditioned on the answer.
Abstract
Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query ($A|Q$). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query ($Q|A$)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments across three datasets spanning code generation (WarriorCoder, Magpie-Qwen2.5-Coder-Pro-300K) and math reasoning (OpenR1-Math-220k) demonstrate that selecting just 25\% of data using stratified RMI matches full-data performance while being consistently competitive with or better than existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability. Code is available at https://github.com/XXSg559/QAQ.
EAHC is proposed, an Execution-Anchored Hallucination Calibration reranking framework that anchors reasoning judgments to execution behavior so that execution-equivalent candidates receive consistent scores, which implements a dual-channel architecture.
Guang Yang, Xing Hu, Xiang Chen et al.· 0 citations
A multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection is presented.
In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions w...
J. Valjakka, Juhani Kivimäki, Juha Mylläri et al.· 0 citations
Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models. We show that the mean displacement from incorrect to correct answer representations, computed at approximately 70\% of model depth from fifty labeled examples with no parameter updates, yields a scoring direction...
M. Armstrong, Navid Ayoobi, Pradham Mummaleti et al.· 0 citations
This work investigates whether an LLM can use an alternative, low level, symbolic competence such as SQL for unsupervised hallucination detection in some high level task and finds that this approach improves on direct prediction and competes with state-of-the-art hallucination detection methods, while not requiring dom...
Renato Vukovic, Hsien-Chin Lin, Carel van Niekerk et al.· 0 citations
It is concluded that RAG should be viewed as a grounding and evidence-access mechanism rather than a guarantee of hallucination-free generation, and applications of RAG are outlined.
Shyalaja L. N., Shantinath Patil, P. R. et al.· International Journal for Re...· 0 citations
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.