ProofTeller: Exposing Recency Bias in LLM Reasoning and Its Side Effects on Communication (Extended Abstract)

Mayank Jobanputra A. Kovtunova Brisca Balthes Fedor Grigoryevich Pogulskiy Yifan Wang Stefan Borgwardt Vera Demberg

2026 · Digital library · 0 citations · 41 references

Computer Science

TL;DR

ProofTeller, a benchmark that evaluates large language models' ability to faithfully interpret its reasoning, finds a consistent near-conclusion bias: LLMs tend to focus on steps closest to the final proof conclusion rather than on the most informative ones.

Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Magenta: Closing the Loop Between Mathematical Reasoning and Lean Verification

This paper introduces Magenta, a training-free agentic pipeline that, given only a natural-language problem, produces an answer, expresses it as a Lean 4 statement, and constructs a machine-checked proof, which achieves 100% accuracy across all evaluated olympiad benchmarks.

Joshua Ong Jun Leang, Hao-Nan Li, Zheng-Yang Zhao et al. · 0 citations

#large language models Open access Sep 2026

Emergent Reasoning in Large Language Models: A Systematic Evaluation Across Task Complexity

It is argued that emergent reasoning in LLMs is a product of three factors: model size, prompting approach, and evaluation metric, and proposed implications for designing benchmarks and assessing capabilities are proposed.

N. Kuotsu · International Journal of Cre... · 0 citations

#artificial intelligence Preprint Aug 2026

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark

DeReLab is introduced, a generative framework that produces multi-turn belief-updating conversations from parameterized graph structures across default and inheritance reasoning, with formally verified ground truth at every turn, enabling controlled measurement of how models respond to confirming and disconfirming evid...

Jayanta Sadhu, S. Shahad, Kenneth Marino · 1 citation

Preprint Aug 2026

When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Models

This paper attempts to develop a formal account of the jump in four steps and measures the second, and hopes it paves the path for formalizing and measuring the full jump in the future.

Dai Shi, Xiao-Yu Li, José Miguel Hernández-Lobato · 0 citations

#artificial intelligence Preprint Aug 2026

Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs

Logical reasoning with large language models (LLMs) is a critical capability, as it reflects a system's ability to correctly deduce hypotheses from a given context using faithful deductive processes. However, LLM reasoning has often been shown to be sensitive to small surface-level variations in problem formulation, ra...

Ramya Keerthy Thatikonda, W. Buntine, Ehsan Shareghi · 0 citations

#machine learning Preprint Aug 2026

When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Models

A formal account of the jump is developed in four steps and measured, proving that jump instances are well-posed and establish a family theorem that certifies instances of unbounded difficulty without enumeration and further formalize when a jump is correct and how successive jumps compound.

Dai Shi, Xiao-Yu Li, José Miguel Hernández-Lobato · 0 citations