Skip to content

The Prompt Is Not the Query: How Request State Evolves Across Multi-Turn AI Conversations

Jul 2026 · arXiv.org · Vol abs/2607.22392 · 2 citations · 16 references
Computer Science

TL;DR

Categorical results support session-level measurement for AI search, and length-matched nulls show that low lexical coverage is largely a consequence of turn length, so vocabulary results are interpreted as information availability, not semantic drift.

Abstract

AI-search evaluation commonly treats a prompt as a stable query that can be counted, classified, and replayed in isolation. A conversation makes that unit of analysis questionable: each user turn can add a constraint, revise an assumption, request evidence, or refer to alternatives established earlier. We replace latent"intent"with an observable construct, conversation-conditioned request state, and measure how that state is distributed across user turns. The analysis reuses frozen rules and the governed cohort of a preceding conversation study: 670 English commercial multi-turn conversations in a discovery-replication design and 7,463 public PRISM conversations from 1,389 participants. In the commercial corpus, the final prompt contains a median 35.6% of the session's unique user-side content vocabulary; in PRISM, the median is 36.4%. The final prompt contains at most half of that vocabulary in 68.4% and 74.3% of conversations, respectively. More importantly, transparent rules detect at least one request-state dimension in history but not in the final prompt in 50.3% of commercial conversations and 44.8% of PRISM conversations. Among dimension-bearing conversations, the final prompt reproduces the full observed dimension set in only 26.1% and 26.2%. At the same time, the final prompt adds a previously unseen dimension in 17.9% and 19.3%, showing that the endpoint is neither a summary nor merely a reference: it is often another state update. Length-matched nulls show that low lexical coverage is largely a consequence of turn length, so vocabulary results are interpreted as information availability, not semantic drift. The categorical results support session-level measurement for AI search. They do not estimate the causal effect of history on model answers.

View source

Similar papers

#natural language process... Preprint Aug 2026

You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding

The results show that CoRG remains challenging for current agents, even the best agent reaches only 67.0% success rate, leaving one third of references unresolved, and position CoRG as a concrete benchmark for studying how agents search, inspect, and verify information in realistic multi-tool environments.

Karen Fuchs, Uri Katz, Yoav Goldberg · 0 citations
Preprint Aug 2026

When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory

The results indicate that for precise, evidence-grounded questions over chat archives, much of the benefit credited to elaborate memory structures is recoverable by giving an agent controllable search over the unmodified record, with no LLM-based index construction at all.

Ruizhe Li, L. Zhang, Benfeng Xu et al. · 1 citation
Preprint Aug 2026

QUMem: Personalized Memory for Query-Conditioned User-State Inference in LLM Agents

A structured memory framework for query-conditioned user-state inference for long-term personalization that achieves state-of-the-art performance on both PersonaMem and KnowU-Bench, demonstrating the effectiveness of query-conditioned user-state inference for long-term personalization.

Heng Wang, Yifei Li, Lingling Zhang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory b...

Wen-Yu Chang, Yun-Nung Chen · 3 citations · ⚡1
Jul 2026

LLMs Get Lost in Evolving User Intent

This work introduces a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns, while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation.

Jihoon Tack, Philippe Laban, Jennifer Neville · 4 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.