Design and Evaluation of a RAG-Based Educational Assistant Grounded in Course Materials: A Case Study in Vocational Training

Abstract

This study presents the design, implementation, and exploratory classroom evaluation of a course-constrained educational assistant combining Retrieval-Augmented Generation (RAG) with large language models (LLMs) in Vocational Education and Training (VET). The platform retrieves teacher-provided materials, generates course-aligned responses, displays source references, and returns a predefined abstention message when evidence appears insufficient. It was deployed during a practical session in the Spanish Higher Vocational Training programme Administración de Sistemas Informáticos en Red (ASIR; Network and Information Systems Administration). Nineteen students and one instructor completed a post-session questionnaire. Students reported positive perceptions of usability, clarity, perceived reliability, and learning support, while restricted corpus coverage and the absence of conversational memory emerged as limitations. To complement the perception-based classroom study, a separate post hoc technical assessment used 30 predefined queries with the same prototype and corpus. All 16 fully answerable prompts received substantive responses; 13 were rated fully correct and three partially correct by the first author. Seven of eight deliberately out-of-corpus prompts triggered abstention, whereas one OpenShift Route query produced an unsupported answer from semantically adjacent but non-supporting Kubernetes evidence. The contribution is empirical and design-oriented rather than algorithmic. The findings are context-specific and show both the potential and limitations of bounded cognitive support grounded in instructor-selected materials.