Skip to content

What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use

Sep 2026 · 0 citations · 34 references
Computer Science

TL;DR

This work shows why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples, and connects the three interventions on the same facts.

Abstract

What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.

View source

Similar papers

#machine learning Preprint Sep 2026

The Token Before the Value Is the Key: How Hybrid Architectures Organize Induction Circuits

Hybrid language models can improve capability as well as efficiency, raising the question of how architectural complementarity becomes learned computation. We examine the established induction roles of Carrying predecessor information, Matching a source by content, and Copying its value. How are these position-sensitiv...

Ke Cheng, Xin Xu, Yi-Xiao Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge

This work compares country-continent questions with noun, adjective, and code answers while keeping several fitted measurements distinct across Qwen, Llama, and Gemma to separate early readability, natural strength, causal steering, and later content dependence.

Wen-Lin Wei, Yuan Fang, Ren-He Jiang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Harness Compilation: Which Decisions Should a Small Vision-Language Model Keep?

Small vision-language models may be able to read external evidence yet struggle to obtain it. We introduce Harness Compilation (HC), an offline procedure that adapts the division of work between a frozen small VLM and its external harness. A large teacher uses student execution traces to revise reusable content and con...

Min-Hao Fan, Yin Liu, Jiayu Zhao et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Decoupling Memory from Context: Structured Memory for Token-Efficient Test-Time Continual Learning

This work introduces a unified formulation of context optimization and shows that an agent memory system update can be interpreted as an optimization update procedure over the model's context, and proposes GraphMemory, a lightweight graph-based memory that accumulates, refines, organizes, and connects reusable strategi...

Yehya Farhat, Michael Desmond, Anastasios Kyrillidis · 0 citations
Preprint Sep 2026

A Historical Corpus Is Not a Historical System: Auditing Hindsight Leakage in Stateful Data Discovery

Offline replay should estimate what a discovery system could retrieve at a historical point, yet freezing the corpus leaves interaction memory unconstrained. We formalize point-in-time (PIT) discovery through historical state $(D_t, \theta_t, M_{<i})$ and introduce a paired replay that changes only memory availability....

Yi-Xi Zhou, Fan Zhang, Si-Kun Wang et al. · 0 citations
#machine learning Open access Sep 2026

Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs

This work provides the first mechanistic base-vs-instruct comparison of conflict-resolution circuits, and believes that because the conflict circuit is preserved rather than rebuilt, interpretability and control tools calibrated on base models should transfer directly to their deployed instruct siblings.

S. Pandere, Gautam Ranka, Ritika Varshney et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.