Extending Search Habits: Unfolding Novice User Experience with Conversational AI through Information Retrieval Tasks

Abstract

As digital libraries and information retrieval (IR) architectures increasingly integrate conversational artificial intelligence (AI) and large language models (LLMs), understanding how pre-existing user cognitive frames shape interaction strategies becomes critical. Traditional user evaluations frequently overlook how novice users’ entrenched mental models formed through exposure to legacy search engines, bibliographic databases, and virtual assistants dictate their prompting behaviors. Utilizing a qualitative empirical study combining interaction logs, cognitive mapping sketches, and semi-structured post-session interviews (N = 7), we investigate how novice users interact with a GPT-4-powered conversational IR interface across diverse information-seeking tasks. We consolidate user conceptualizations into three primary mental model typologies: Database/Retrieval, Conversational/Synthesis, and Skeptical/Verification, mapping them to behavioral instructional hierarchies and prompt categorization frameworks. Findings reveal that extensive database experience predicts severe cognitive rigidity (functional fixedness), while age plays a secondary role. We conclude with implications for digital library architectures, advocating for transparent provenance, adaptive scaffolding, and XAI onboarding mechanisms.