DEVELOPMENT OF A RAG-BASED SCHOLARLY RESEARCH CHATBOT WITH FINE-TUNED DENSE RETRIEVAL

Abstract

Foundation models, particularly Large Language Models (LLMs), show promise for chatbots for scholarly research chatbots but suffer from “hallucinations” and limited domain knowledge. This work addresses these reliability issues by developing a Retrieval-Augmented Generation (RAG) chatbot designed explicitly for scholarly research interactions within the Natural Language Processing (NLP) domain. The system utilizes a domain-specific corpus from the Association of Computational Linguistics (ACL) Anthology. The corpus was processed into text chunks and indexed using a high-dimensional vector database to facilitate efficient information retrieval. The generation component comprises an instruct-tuned LLM; Mixtral 8x7b, which synthesizes the retrieved context into coherent, evidence-based responses while maintaining the original semantic integrity of the scholarly source. Retrieval is performed using semantic similarity search. To improve dense retrieval precision, the BAAI/bge-large Sentence Transformer model was fine-tuned on a synthetic dataset generated from the corpus. Results show that fine-tuning and integrating the BAAI/bge-large model significantly improves the chatbot's ability to retrieve relevant information. Compared to the base model, retrieval accuracy of the fine-tuned model increases by up to 15% across different metrics: Accuracy@k, Precision@k, Recall@k, where k represents the count of retrieved chunks considered for evaluation, and Mean Reciprocal Rank (MRR@10). Notably, accuracy reaches an impressive 97%, demonstrating a significant boost in retrieving the most relevant scholarly information for user queries. These findings underscore the efficacy of fine-tuned Retrieval-Augmented Generation (RAG) systems in developing reliable, grounded chatbots for academic environments.