Speculative Response Generation for Live Cross-Language Dialogue: Hiding Language-Model Latency Inside the Interlocutor's Turn
Language models can formulate conversational responses, but their inference latency — typically measured in seconds — is incurred at exactly the wrong moment: after an interlocutor finishes speaking, when the socially acceptable response gap is shortest. This paper describes a mechanism that relocates that latency. Whi...