Question Types for Knowledge Acquisition in LLM-based Dialogue Systems: Experiments with Simulated and Human Users
TL;DR
The findings suggest that candidate-guided question forms are useful when the system has a plausible candidate answer, whereas wh-questions may be appropriate when the system lacks sufficient confidence to ask a more specific question.
Abstract
To acquire knowledge from users through dialogue, systems must decide not only what to ask but also how to ask it. Since users may not always be willing to answer such questions, question formulation affects both user experience and the information obtained. Prior work has examined question types in controlled, template-based settings, but it remains unclear whether similar effects are observed in LLM-based dialogues. We investigated the effects of question type on knowledge acquisition through experiments with both an LLM-based user simulator and crowdsourced human participants. We compared three question types: implicit questions, explicit questions, and wh-questions. The results showed no substantial differences in user annoyance among the question types. In contrast, the question types differed in how well they elicited correct responses: wh-questions were less effective, whereas implicit and explicit questions performed comparably. The findings suggest that candidate-guided question forms are useful when the system has a plausible candidate answer, whereas wh-questions may be appropriate when the system lacks sufficient confidence to ask a more specific question.