Conversational Style in Open Domain Dialogue Systems: What Makes a Response Sound Natural

Abstract

Intelligent Virtual Agents need to be able to participate in extended dialogue interactions while maintaining a conversational style. We discover the elements of conversational style in open-domain dialogues by analyzing the features that distinguish conversational system responses from responses that are merely topically relevant. We first collect 7,751 dialogue contexts from live human conversations with a multi-generator Alexa Prize SocialBot that was deployed across four competition years. We also collect 35,623 candidate responses for the contexts. We annotate the responses with a four-level ABCD quality scheme that isolates conversational naturalness from topical relevance. We then extract twenty-five linguistic features that capture conversational properties of responses and contrast (A) responses that have a conversational style, from (B) responses that are topically relevant, but less conversational. We find that conversational responses are marked by other-directed engagement: question-asking, user engagement phrases, acknowledgment openings, and second-person reference, while responses that are merely relevant and topical are marked by self-oriented information delivery: opinion markers, formulaic openings, and hedges and emphasizers deployed in service of the system’s own assertions. We thus find that conversational style in this setting is best understood as a pragmatic orientation toward the user rather than toward the system’s own content, and that the system must be mixed-initiative to manifest a conversational style. We discuss what these findings imply for the design of Intelligent Virtual Agents.