Adaptive Personalisation for Effective Conversational Search Systems

Abstract

Conversational Information Seeking (CIS) requires systems to maintain context across turns, resolve referential ambiguities, and adapt to evolving user intent [6]. Beyond contextual understanding, truly effective CIS demands personalisation: adapting retrieval and responses to each user's unique background and preferences [2]. However, two fundamental limitations constrain current systems. First, existing conversational search pipelines apply a single static workflow to every query, lacking the adaptability to decide when personalisation helps and when it degrades retrieval precision. Second, personalisation in standard benchmarks typically relies on clean, hand-authored oracle user profiles, ignoring the reality that real-world user context is buried within noisy, long-term interaction logs. This research addresses both limitations by proposing a novel modular, agentic framework, realised through a suite of specialised agents that are coordinated by a dynamic orchestration mechanism. The first completed contribution (RQ1) addresses the granularity of personalisation during query reformulation. We introduced a new set of human annotations for the iKAT 2023 test collection to capture the specific personalisation requirements of queries. Leveraging this data, we developed an adaptive framework with a query-level classifier that determines whether the retrieval quality of a given conversational query can be improved by including the user profile. Consequently, queries are selectively routed between a general and a personalised reformulation path, preventing profile noise from degrading performance on simple information needs [4]. The second completed contribution (RQ2) extended this adaptive personalisation ability to the ranking phase. We developed an adaptive reranking framework that dynamically routes queries between standard similarity-based rerankers and specialised instruction-following rerankers based on conversational intent [3]. When complex, user-specific constraints are detected, the system routes the retrieved documents to an instruction-following reranker. This reranker reorders the documents according to nuanced, profile-derived instructions, bridging the gap between raw retrieval and generation-ready contexts [5]. Current work (RQ3) targets the oracle-profile assumption. Rather than relying on static, pre-defined user profiles assumed at test time, we propose an agentic long-term memory component that dynamically builds and maintains personalised user profiles from noisy historical interaction logs. We evaluate this component using the TREC iKAT test collections to determine whether dynamically extracted memory can approach static oracle profiles performance without requiring manually curated profiles, which are costly to construct and maintain. As future work, I plan to continue this research in two key directions. I will integrate this dynamically extracted memory into personalised response generation (RQ4), evaluating completeness, groundedness, and naturalness via LLM-as-a-judge [2] and nugget-based evaluation frameworks [1]. In addition, to conclude the thesis statement validation, I will explore a dynamic workflow generator (RQ5) that adaptively generates effective and efficient workflows for orchestrating specialised agents based on query complexity, long-term memory about the user and conversational context, aiming to enhance the retrieval effectiveness of static multi-agent pipelines.