Skip to content

QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions

Mar 2026 · arXiv.org · Vol abs/2603.12165 · 0 citations · 36 references
Computer Science

TL;DR

QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query conditioned on the query, is proposed and Reverse Mutual Information (RMI) is defined to quantify the information gain about the query conditioned on the answer.

Abstract

Synthetic data has become essential for training code generation models, yet it introduces significant noise and hallucinations that are difficult to detect with current metrics. Existing data selection methods like Instruction-Following Difficulty (IFD) typically assess how hard a model generates an answer given a query ($A|Q$). However, this metric is ambiguous on noisy synthetic data, where low probability can distinguish between intrinsic task complexity and model-generated hallucinations. Here, we propose QAQ, a novel data selection framework that evaluates data quality from the reverse direction: how well can the answer predict the query ($Q|A$)? We define Reverse Mutual Information (RMI) to quantify the information gain about the query conditioned on the answer. Our analyses reveal that both extremes of RMI signal quality issues: low RMI indicates semantic misalignment, while excessively high RMI may contain defect patterns that LLMs easily recognize. Furthermore, we introduce a selection strategy based on the disagreement between strong and weak models to identify samples that are valid yet challenging. Experiments across three datasets spanning code generation (WarriorCoder, Magpie-Qwen2.5-Coder-Pro-300K) and math reasoning (OpenR1-Math-220k) demonstrate that selecting just 25\% of data using stratified RMI matches full-data performance while being consistently competitive with or better than existing data selection methods. Our approach highlights the importance of bidirectional semantic coherence in synthetic data curation, offering a scalable pathway to reduce computational costs without sacrificing model capability. Code is available at https://github.com/XXSg559/QAQ.

View source

Similar papers

Preprint Aug 2026

Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation

EAHC is proposed, an Execution-Anchored Hallucination Calibration reranking framework that anchors reasoning judgments to execution behavior so that execution-equivalent candidates receive consistent scores, which implements a dual-channel architecture.

Guang Yang, Xing Hu, Xiang Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Domain-Specific Hallucination Detection in Large Language Models

A multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection is presented.

Varun Teja Chundru, Debasmita Biswas · 0 citations
#natural language process... Preprint Oct 2026

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions w...

J. Valjakka, Juhani Kivimäki, Juha Mylläri et al. · 0 citations
#natural language process... Preprint Oct 2026

Correctness Is a Direction: Geometric Answer Selection in Language Models

Answer correctness is encoded as a recoverable geometric direction in the hidden states of language models. We show that the mean displacement from incorrect to correct answer representations, computed at approximately 70\% of model depth from fifty labeled examples with no parameter updates, yields a scoring direction...

M. Armstrong, Navid Ayoobi, Pradham Mummaleti et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Leveraging Low-Level Symbolic Competences for Unsupervised Grounding in Hallucination Detection

This work investigates whether an LLM can use an alternative, low level, symbolic competence such as SQL for unsupervised hallucination detection in some high level task and finds that this approach improves on direct prediction and competes with state-of-the-art hallucination detection methods, while not requiring dom...

Renato Vukovic, Hsien-Chin Lin, Carel van Niekerk et al. · 0 citations
Review Open access Aug 2026

Large Language Models Hallucinate and How Retrieval- Augmented Generation Mitigates It

It is concluded that RAG should be viewed as a grounding and evidence-access mechanism rather than a guarantee of hallucination-free generation, and applications of RAG are outlined.

Shyalaja L. N., Shantinath Patil, P. R. et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.