Evaluating Retrieval-Augmented Generation Strategies with a Virtual Human Application
TL;DR
The described Virtual Human demonstration system uses RAG with 14 geriatrics patient brochures from the Canisius Wilhelmina Hospital, which were converted into question-answer chunks and embedded and retrieved based on similarity to user queries.
Abstract
Virtual Humans enhanced with Large Language Models can hold broad conversations, but their answers may sound convincing while still being factually incorrect. Such hallucinations can mislead users and reduce trust, especially because research shows that people often overestimate LLM accuracy and may remain distrustful after errors occur. This loss of trust is particularly concerning in medical contexts, where reliable information is essential. Retrieval-augmented generation (RAG) addresses this issue by grounding LLM responses in external knowledge sources that were not part of the model's training data. The described Virtual Human demonstration system uses RAG with 14 geriatrics patient brochures from the Canisius Wilhelmina Hospital, which were converted into question-answer chunks and embedded. At runtime they were retrieved based on similarity to user queries. The system allows for comparison of four RAG approaches, i.e. Basic RAG, HyDE RAG, Reranking RAG, and Cognitive RAG, each using different strategies to improve retrieval and response quality. Users can choose among these RAG options in the application menu, as well as set some other user preferences. Response quality can be evaluated using RAGAS metrics and latency related data. User experience data can also be collected for correlation to RAG performance data.