Development of a Retrieval-Augmented Generation Chatbot for Academic Regulation Information Services

Abstract

Academic information services in higher education institutions still face challenges because important information is often distributed through static, lengthy, and difficult-to-navigate PDF regulatory documents. This condition makes it difficult for students to obtain specific academic information efficiently and may increase repetitive inquiries directed to academic administrative staff. This study aims to develop and evaluate a Retrieval-Augmented Generation (RAG)-based chatbot for academic regulation information services using a locally deployed Large Language Model (LLM). The main contribution of this study lies in the implementation of a local RAG architecture for hierarchical academic regulation documents by combining hybrid chunking, hybrid retrieval, reranking, and evaluation using retrieval metrics and RAGAS. The proposed method includes document preprocessing, document segmentation with variations in chunk size and chunk overlap, embedding generation, storage in ChromaDB, hybrid retrieval combining semantic and lexical search, and integration of Gemma 3:4B through Ollama. System evaluation was conducted through six experimental scenarios with different retrieval configurations. Retrieval performance was measured using Precision, Recall, and Mean Reciprocal Rank (MRR), while response quality was evaluated using Faithfulness, Answer Relevancy, Context Precision, and Context Recall. The best configuration was achieved with Top-K 10, chunk size 1024, and chunk overlap 400, producing scores of 0.7950, 0.7868, 0.8093, and 0.8958, respectively. These results indicate promising feasibility for supporting document-based academic regulation information services, although broader validation is still required due to the limited evaluation dataset and restricted document scope.