Skip to content

The Key Handoff: Retrieval in Hybrid Language Models

Sep 2026 · 0 citations · 31 references
Computer Science

TL;DR

Across twelve models, dense and hybrid, the authors move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives, which makes the answer a handoff.

Abstract

A two-hop question makes a language model retrieve twice: once to produce a bridge entity, and once to retrieve with it. Transformers resolve that entity in their early layers. Hybrid models replace most of the attention with a recurrent state, so where the key becomes usable, and where it is spent on an answer, is not known. Answering"Where is the ball?"from"the ball belongs to Alice"and"Alice is in the garden"turns on Alice, a name the question does not mention. A model could reach garden through Alice, the key it computed, or through where the fact sits in the prompt. Across twelve models, dense and hybrid, we move a hidden state from one story into another where the two routes lead to different places, and read off which place the model gives. An attention layer converts the key in every model we tested, and crossing that layer removes its usable effect, all of it where no attention follows. In sequential hybrids this makes the answer a handoff: recurrent layers carry the key forward, and attention spends it. Retrieval does not always stop there. Writing a different fact into the memory of the recurrent layers that follow the last attention layer can move the answer toward that fact, multiplying its odds by 1.3 to 2.7, and which hybrids do this is not settled by their architecture or training. That read is addressed by the key, not by position: recurrent state in a hybrid is not only a carrier, but a memory that later layers can query.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

From Parameters to Answers: How LLMs Retrieve and Use Their Internal Knowledge

This work compares country-continent questions with noun, adjective, and code answers while keeping several fitted measurements distinct across Qwen, Llama, and Gemma to separate early readability, natural strength, causal steering, and later content dependence.

Wen-Lin Wei, Yuan Fang, Ren-He Jiang et al. · 0 citations
#natural language process... Open access Oct 2026

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons...

Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense · 0 citations
#natural language process... Preprint Sep 2026

Don't Lose Entities from Retrieval to Generation: Dual Entity Recovery RAG for multi-hop QA

Retrieval-augmented multi-hop question answering (QA) decomposes a query into sub-questions and decomposes the corpus into smaller retrieval units such as sentences. Both forms of decomposition improve the pipeline, but we show that both share the same vulnerability, the loss of entity information, and that this loss b...

Heechan Lee, Dong-Young Lim · 0 citations
#artificial intelligence Preprint Sep 2026

Language Models Can Control Their Own Attention

Declarative Attention is introduced, a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes:(full context),(a specific region), and(recent output only).

Namgyu Ho, Huzama Ahmad, Woosung Koh et al. · 1 citation

Where a Model Sends Its Own Repeated Token

Black-box model identification works by scoring a model's response to natural-language prompts. One line of work feeds models a degenerate input -- their own token, repeated -- to find a failure mode rather than an identity. We take that input and ask where the model goes when it does not. For each token t, read argmax...

Nicolás Vera Zúñiga · 0 citations
#artificial intelligence Preprint Sep 2026

Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/without comparison cannot say which part of the layer does the work: the semantic content, the retrieval scaffolding that delivers it, or the pre-computed views that usuall...

Qing Ye · 0 citations

Related blog posts

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.