Optimizing RAG-Based Academic Chatbot Performance Using Prompt Engineering

Optimizing Context Injection for Educational Chatbots through Comparative Sentence and Recursive Chunking Strategies

This research addresses the common challenge of a lack of context in Question and Answer (QA) datasets in digital education, which limits the reasoning potential of Large Language Models (LLMs). To address this, we optimize an automated retrieval-based dataset generation system that systematically enriches QA pairs with relevant pedagogical context from authoritative digital textbooks. This study conducts a comparative analysis of two major text chunking strategies: sentence chunking and recursive chunking. Although these pipelines are designed for general education applications, they are evaluated here through a case study of Indonesian elementary education materials. To ensure the highest reliability, the workflow performance is measured against a ground truth dataset of 978 entries, manually curated and validated by education experts to ensure pedagogical accuracy, and 781 entries from other subjects. Quantitative evaluation using BERTScore shows that recursive chunking achieves a superior F1 score of 0.748 compared to 0.737 for sentence chunking, with peak performance observed on upper elementary school materials (Grades 5 and 6). These findings were corroborated by the final verification phase through User Acceptance Testing (UAT) with an elementary school educator, where recursive chunking achieved a 'Relevant' score of 22 compared to 17 for sentence chunking. A key contribution of this study is the development and validation of a standardized, automated workflow by experts that effectively overcomes the barriers of manual dataset construction for domain-specific tasks, providing a semantically robust foundation for context-aware educational AI.

RAG-test: a framework to evaluate RAG chatbot performance and measure productivity gains

Context-Aware Large Language Model for Customer Support Chatbots

Despite their high conversational fluency, large language models (LLMs) tend to produce responses that are either contrived or factually inaccurate, a phenomenon known as hallucination. This constrains their dependability in areas where accuracy is crucial, like customer service. This study leverages a context-aware chatbot built on a Retrieval-Augmented Generation (RAG) pipeline to solve the problem. The system retrieves semantically relevant text from an external knowledge base by integrating neural information retrieval with LLMs to ensure factual correctness and enhanced contextual relevance. These externally retrieved documents are given as a reference during response generation. The LLM-as-a-judge approach is used to evaluate the system by comparing responses to a qualitative performance matrix using GPT-4o. Results show that the RAG-based chatbot increases context precision by a factor of 7.5, decreases hallucinations (measured through Faithfulness) by 73.20% and improves answer relevance by 6.97% when compared to a random retrieval baseline system. This study advances conversational AI by demonstrating how the retrieval method significantly enhances the usefulness and reliability of LLMs for enterprise-level customer service. The results show that the RAG architecture provides a scalable alternative for creating precise, contextually grounded conversational agents, thereby mitigating some of the main drawbacks of LLMs.

DEVELOPMENT AND ACCEPTABILITY OF AN INTELLIGENT CHATBOT FOR STUDENT SERVICES IN EASTERN SAMAR STATE UNIVERSITY SALCEDO CAMPUS

The study successfully validated that the intelligent chatbot efficiently bridges communication barriers, automates repetitive administrative inquiries, and improves service accessibility, confirming that the application is highly usable, practical, and effective as an inclusive, assistive communication tool.

Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning

Large language models (LLMs) deployed in educational settings often behave as direct answerers: they disclose target concepts in the opening turn instead of guiding students through progressive inquiry, as Socratic pedagogy prescribes. We present HeuristicEdu, a two-phase pipeline that aligns Qwen2.5-7B toward Socratic tutoring via supervised warm-up and Group Relative Policy Optimization (GRPO). Training uses SocraticEdu, 797 multi-turn Chinese children's science dialogues reconstructed from a live platform, with a heuristic reward over cognitive depth (R_cog), curiosity engagement (R_eng), and directness (R_dir), together with a K_query correction for student-introduced terms. We introduce Scaffolding Effectiveness (SE) and Conversation Depth (CD) to evaluate outcomes beyond surface fluency. On 30 held-out questions, the best GRPO variant improves SE from 30.0% to 63.3% and lowers keyword leakage from 30.0% to 13.3%. Notably, this best variant omits the directness penalty during optimization, suggesting that explicit anti-leakage terms can conflict with gradient-based behavioral alignment. An unaligned Qwen-72B baseline reaches 0% SE and 96.7% leakage, showing that scale alone does not induce Socratic behavior.

An On-Premise Multilingual Academic Chatbot using Retrieval-Augmented Generation and Context-Aware Memory for University Assistance

Universities now use Large Language Models (LLMs) to transform their processes for managing student information. The paper introduces an upgraded chatbot system for Narasaraopeta Engineering College (NEC) which extends previous on-premise LLM chatbot research by providing four new functions. The system uses (1) Retrieval-Augmented Generation (RAG) to create citation-based responses through LlamaIndex and ChromaDB, (2) Context Memory which maintains conversation flow during multiple dialogue exchanges, (3) Voice Input through OpenAI Whisper Speech-to-Text (STT) technology, and (4) Multilingual Support which covers English and these seven languages: Hindi, Telugu, Tamil, Kannada, and Malayalam through IndicNLP. The system tested 60 benchmark questions across four academic categories which included regulations and examination policies and fee structures and multilingual queries and achieved 96.7% overall accuracy with sub-second text response times and 1.0–1.4 second voice response times. The system operates entirely on-premise through Docker which safeguards institutional data privacy while eliminating the need for recurring cloud API expenses. The upcoming development will create Emotion-Aware AI, FAQ Auto-Learning, Student Portal Integration, and a Mobile Application.