Speculative Response Generation for Live Cross-Language Dialogue: Hiding Language-Model Latency Inside the Interlocutor's Turn

Abstract

Language models can formulate conversational responses, but their inference latency — typically measured in seconds — is incurred at exactly the wrong moment: after an interlocutor finishes speaking, when the socially acceptable response gap is shortest. This paper describes a mechanism that relocates that latency. While the interlocutor is still speaking, the system consumes streaming partial speech-recognition hypotheses — incomplete, incrementally updated transcriptions carrying word-level timestamps and confidence scores — together with the dialogue history and a persistent user profile, and generates a small set of candidate responses in the interlocutor's language in parallel with the ongoing speech. Each significant partial-hypothesis update re-triggers generation; candidates are refined, stale candidates are replaced, and the set is re-ranked, so that it converges as the turn's semantics accumulate and a final ranked set is ready before the listener's turn begins. A scoring function ranks candidates by contextual relevance, user-profile match, and predicted conversational utility, and a confidence gate suppresses presentation until the top candidate's combined score exceeds a configurable threshold, leaving the display in subtitle-only mode rather than distracting the listener with low-confidence guesses. The paper sets out the pipeline, the convergence behaviour that distinguishes the mechanism from one-shot reply suggestion, the ranking and gating design, and the disclosed alternative embodiments, including on-device generation with a distilled model and automatic identification of the target language. The mechanism is described as disclosed in U.S. Provisional Patent Application No. 64/170,247; no performance measurements are reported, and numerical values cited are the examples and ranges stated in that disclosure.