The system offers a replicable instantiation of RAG for transparent, citation-backed campus information services in Indonesian and offers a replicable instantiation of RAG for transparent, citation-backed campus information services in Indonesian.
Abstract
University service centers handle many repetitive queries about academic schedules, registration, and policies stored in internal documents. Manual lookup is inefficient, and answers given by staff can be inconsistent. Rule-based chatbots only handle limited question patterns, while large language models are hard to update and may produce unsupported answers (hallucinations). This research designs an academic chatbot that combines document retrieval with answer generation so that each answer remains traceable to its source. The system extracts text from campus documents, segments it, encodes it using a multilingual embedding model, and stores it in a vector index for context retrieval. A response is generated through an instruction template that confines the output to the retrieved information and includes page references. Evaluation followed a mixed-method design: a quantitative layer measured retrieval quality (Precision@5, Recall@5) and generation quality using the four RAGAS sub-metrics (faithfulness, answer_relevancy, context_precision, context_recall) on a 100-question test set, while a qualitative layer applied thematic analysis to open-ended user comments. Statistical testing used McNemar's test for accuracy and a paired bootstrap (10,000 resamples) for retrieval metrics; 95% confidence intervals are reported. Results: the proposed RAG system achieved 84% answer accuracy (95% CI 76–90%), Precision@5 = 0.80 and Recall@5 = 0.72, with a System Usability Scale (SUS) score of 78 and a Net Promoter Score (NPS) of +32 from 30 participants. Differences in accuracy versus the lexical and LLM-only baselines were statistically significant (McNemar p < 0.05). The system offers a replicable instantiation of RAG for transparent, citation-backed campus information services in Indonesian.
Academic information services in higher education institutions still face challenges because important information is often distributed through static, lengthy, and difficult-to-navigate PDF regulatory documents. This condition makes it difficult for students to obtain specific academic information efficiently and may increase repetitive inquiries directed to academic administrative staff. This study aims to develop and evaluate a Retrieval-Augmented Generation (RAG)-based chatbot for academic regulation information services using a locally deployed Large Language Model (LLM). The main contribution of this study lies in the implementation of a local RAG architecture for hierarchical academic regulation documents by combining hybrid chunking, hybrid retrieval, reranking, and evaluation using retrieval metrics and RAGAS. The proposed method includes document preprocessing, document segmentation with variations in chunk size and chunk overlap, embedding generation, storage in ChromaDB, hybrid retrieval combining semantic and lexical search, and integration of Gemma 3:4B through Ollama. System evaluation was conducted through six experimental scenarios with different retrieval configurations. Retrieval performance was measured using Precision, Recall, and Mean Reciprocal Rank (MRR), while response quality was evaluated using Faithfulness, Answer Relevancy, Context Precision, and Context Recall. The best configuration was achieved with Top-K 10, chunk size 1024, and chunk overlap 400, producing scores of 0.7950, 0.7868, 0.8093, and 0.8958, respectively. These results indicate promising feasibility for supporting document-based academic regulation information services, although broader validation is still required due to the limited evaluation dataset and restricted document scope.
This paper presents the design and implementation of intelligent chatbots using Retrieval-Augmented Generation architecture to derive accurate, context-based responses from domain-specific knowledge sources such as PDFs and CSV files. The proposed system integrates information retrieval mechanisms with large language models to overcome crucial limitations of traditional chatbots: hallucination, knowledge cut-off, and inability to access private organizational data. The proposed architecture uses a six-stage pipeline consisting of document ingestion, text splitting, and embedding generation with sentence transformers, semantic retrieval, and response generation via Large Language Models (LLMs). By preprocessing the PDF and CSV documents using a RecursiveCharacterTextSplitter and converting them into 384-dimensional semantic embeddings, the system works via a similarity-based retrieval of document chunks. For implementation, the LlamaIndex framework was used for data indexing and query orchestration, and HuggingFace Transformers for language modeling. Experimental validation conducted on the smart campus chatbot system demonstrated significant improvements in the accuracy, relevance and transparency of responses across education, student services, and administrative support domains compared with a standalone language model. Because of its modular design, it supports a wide range of domain-specific topics without the combinatorial costs of re-training. This is another important application for systems that require regular updates and accurate information. Overall, the proposed framework provides a simple and practical approach for easily implementing domain-aware chatbots.
Varsha Spoorthi Shree Narravula, V. K, Yashaswini B. Suresh et al.· 2026 7th International Confe...· 0 citations
Academic service units need question-answering systems that respond quickly while remaining aligned with institutional regulations. This study provides a controlled, deployment-oriented comparison of a lightweight FAQ chatbot and a Retrieval-Augmented Generation (RAG) chatbot for academic-service question answering. The novelty lies in evaluating both prototypes against the same version-controlled official-document scope, the same 60-question dataset, category-level service cases, paired statistical tests, inter-rater reliability, and RAGAS retrieval-quality metrics. The questions covered registration, KRS/KHS, schedules, tuition payment, thesis or final project, and administrative letters. Three validators assessed accuracy, relevance, response effectiveness, and response time using official documents as the reference standard. RAG achieved 88.33% accuracy, a mean relevance score of 4.48/5, and 86.67% response effectiveness; FAQ achieved 68.33%, 3.46/5, and 65.00%, respectively. Paired tests confirmed significant advantages for RAG in accuracy, relevance, and effectiveness, while FAQ was significantly faster (0.62 s vs 2.41 s). RAGAS evaluation showed context precision of 0.86, context recall of 0.82, faithfulness of 0.89, and answer relevancy of 0.91. The findings support a cautious hybrid strategy: FAQ for stable repetitive requests and RAG for contextual, document-dependent questions, subject to broader testing with live queries and stronger retrieval baselines.
Suherwin Suherwin, M. Zainuddin, Rachmat Rachmat· G-Tech· 0 citations
Public service chatbots are expected to deliver recommendations from an underlying public service directory, while also making sure that the recommendations respect explicit user constraints. In practice, public service directories are noisy and inconsistent, and general-purpose large language model (LLM) or AI-based chatbots frequently generate unreliable recommendations, citing unverified sources from the web. We investigate the impact of retrieval quality on constraint-aware recommendation in public service conversational systems built over noisy and heterogeneous service directories. We propose TRACE (Trustworthy Retrieval-Augmented Conversational Engine), a retrieval-based, constraint-aware framework that parses input user queries into structural and semantic constraints for downstream retrieval, with the help of a dual data representation schema. Using a curated statewide pantry directory and a synthetic query benchmark, we evaluate multiple knowledge-representation variants with and without knowledge graphs (KGs). We experiment with several open-source LLMs and a proprietary model, showing that strengthening retrieval substantially improves user constraint satisfaction while reducing hallucinated recommendations. Performance differences across LLMs narrowed in our experiments as retrieval quality improved, making results less sensitive to model size. These findings suggest that the quality of retrieval is key for robust public service conversational systems.
Despite their high conversational fluency, large language models (LLMs) tend to produce responses that are either contrived or factually inaccurate, a phenomenon known as hallucination. This constrains their dependability in areas where accuracy is crucial, like customer service. This study leverages a context-aware chatbot built on a Retrieval-Augmented Generation (RAG) pipeline to solve the problem. The system retrieves semantically relevant text from an external knowledge base by integrating neural information retrieval with LLMs to ensure factual correctness and enhanced contextual relevance. These externally retrieved documents are given as a reference during response generation. The LLM-as-a-judge approach is used to evaluate the system by comparing responses to a qualitative performance matrix using GPT-4o. Results show that the RAG-based chatbot increases context precision by a factor of 7.5, decreases hallucinations (measured through Faithfulness) by 73.20% and improves answer relevance by 6.97% when compared to a random retrieval baseline system. This study advances conversational AI by demonstrating how the retrieval method significantly enhances the usefulness and reliability of LLMs for enterprise-level customer service. The results show that the RAG architecture provides a scalable alternative for creating precise, contextually grounded conversational agents, thereby mitigating some of the main drawbacks of LLMs.
Rabia Shabbir, K. Talpur, Shakeel Ahmad· ICCK Transactions on Machine...· 0 citations
Due to the surge in digital document creation, intelligent systems that can effectively retrieve accurate information from unstructured PDF content have become a necessity. Current keyword-based search techniques lack semantic meaning, while large language models (LLMs), when utilized individually, produce hallucinated results when they are not contextually aware of document content. In this paper, we introduce a novel real-time question answering system for PDF content using a lightweight approach to Retrieval-Augmented Generation (RAG). Our system combines a HuggingFace sentence transformer model for semantic embeddings with a FAISS vector similarity search for efficient retrieval, leveraging a Groq-hosted LLaMA 3.3 70B model for context-grounded answer generation. The system is built using LangChain for orchestration and Streamlit for web-based deployment, and requires no GPU, no fine-tuning, and no pre-indexed corpus. Evaluation across five complementary metrics—Exact Match, Token-level F1, BLEU, ROUGE, and semantic similarity—demonstrates strong alignment between generated and reference answers, with a peak semantic similarity of 0.8709 and a mean Token F1 of 0.4701, confirming the viability of this approach for real-world document intelligence.
Mukesh Lakshmi Sai Medikonda, Kalva Vishnu Teja, K. Greeshma et al.· International Conference Com...· 0 citations