Skip to content

REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing

Sep 2026 · arXiv.org · Vol abs/2609.36984 · 0 citations · 34 references
Computer Science

TL;DR

Results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning, and introduce REALHOP, a diagnose-construct-verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations.

Abstract

Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commonly evaluate this ability with questions built around predefined reasoning chains, treating a correct answer as evidence that the intended composition was used. Yet answer correctness alone leaves open whether success depends on the evidence associated with each intended step: models may instead rely on memorized associations, shorter paths, or partial evidence. We examine this dependence using the Behavioral Necessity Rate (BNR), which measures how often targeted evidence removal prevents answer recovery on initially correct instances. Across five existing benchmarks, panel-mean BNR ranges from 16.6% to 48.9%, exposing a substantial gap between annotated structure and observed dependence. Guided by this diagnosis, we introduce REALHOP, a diagnose-construct-verify framework that rebinds entities, factorizes selected relations, adds complete competing paths, and places evidence at traceable locations. Structural and semantic checks precede freezing; behavioral interventions follow. On 790 paired MuSiQue questions, REALHOP raises panel-mean BNR from 27.4% to 94.4% while retaining high Full accuracy. It also yields high BNR on REALHOP-FRAMES and REALHOP-LONGBENCH. On 216 long-context questions, the matched multiple-choice spread across 16 models grows from 13.9 to 59.2 points and persists under repeated open-ended evaluation. Together, these results show that a conceptually coherent chain and a correct final answer do not by themselves establish multi-hop reasoning. Verifying that success depends on every intended hop is therefore as fundamental to multi-hop evaluation as measuring answer accuracy itself.

View source

Similar papers

Are Stated Reasoning Steps Causally Load-Bearing?

This work uses synthetic multi-hop lookup tasks to measure faithfulness causally at the activation level, specifically on self-generated reasoning, and aims to measure faithfulness causally at the activation level, specifically on self-generated reasoning.

Abhiram Bhupatiraju, Rayan Nyaupane · 0 citations
#small language model Preprint Oct 2026

COMPASS: Finding Where Reasoning Lives in Language Models

Comparing three model families and multiple math benchmarks, COMPASS outperforms the activation-steering baselines the authors compare against, improves GSM8K accuracy by 16 percentage points on average, and approaches CoT accuracy with 20-70\% fewer generated tokens.

Pratyay Dutta, Kowshik Thopalli, V. Narayanaswamy · 0 citations
#artificial intelligence Preprint Aug 2026

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark

DeReLab is introduced, a generative framework that produces multi-turn belief-updating conversations from parameterized graph structures across default and inheritance reasoning, with formally verified ground truth at every turn, enabling controlled measurement of how models respond to confirming and disconfirming evid...

Jayanta Sadhu, S. Shahad, Kenneth Marino · 1 citation
#artificial intelligence Preprint Oct 2026

Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability

Chain-of-thought (CoT) reasoning allows humans to inspect how large language models reach their answers, and oversee model behaviour. This reasoning comes at an increased inference cost, motivating efficient methods that train models to solve tasks using fewer tokens. However, a common concern is that such training may...

Samuel Lewis-Lim, Xing-Wei Tan, Mario Sanger et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Evidence-First Agentic Verification

Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the execution trace. A judge may overlook this evidence or accept unsupported claims, including an agent's own reports of success. When rubric v...

Reza Sanayei, Razvan-Gabriel Dumitru, Sun-Jie Hou et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.