An academic chatbot with hybrid retrieval and local large language models

Abstract

Higher-education institutions devote a substantial share of their staff time to answering queries whose answers are already set out in institutional regulations. This paper presents the design and evaluation of an academic chatbot based on retrieval-augmented generation that lets users query those documents in natural language and receive grounded answers in Spanish. The system combines a structure-aware ingestion pipeline, a hybrid retriever that fuses dense semantic search and lexical search through reciprocal rank fusion, and a locally hosted large language model. A distinguishing goal is data sovereignty: every component runs on institutional hardware, so no data is sent to external services. Quality is evaluated with the RAGAS framework over five regulatory documents: the corpus-wide set attains a global score of 0.90, with faithfulness of 0.97. The acceptance criterion is not met, however, under its conjunctive reading: answer relevancy stays below its threshold across all sets, and p95 latency reaches 6.47 seconds against a 5-second target. Quality further degrades on long, heavily cross-referenced regulations. A contrast questionnaire administered to nineteen users places the perception index at 3.64 out of 5 and exposes a relevant discrepancy: only 57.9% agree that the system does not produce fabricated information, against the faithfulness the automatic judge assigns it.