Lightweight Llama Models with Experts’ Curated RAG for Electrical Engineering Education: An Exploratory Comparison

Designing Reliable LLM Systems for AI-Assisted Engineering Education

Large Language Models (LLMs) are increasingly used in AI-assisted engineering education, enterprise learning, technical knowledge support, and software-development workflows. However, static retrieval-augmented generation (RAG) pipelines retrieve external context for every query, which can increase latency, token cost, and irrelevant-context exposure, while direct generation can produce unsupported or stale answers for policy-sensitive, documentation-dependent, or multisource questions. This paper presents a production-oriented adaptive retrieval framework that treats retrieval as a conditional reliability decision rather than a default execution step. The framework combines query embeddings, similarity-based query memory, feedback-confirmed routing labels, comparison/fallback handling, and observability logs to decide whether a query should be answered directly, routed through retrieval, or escalated for evidence comparison. The system was evaluated in a production-similar controlled test bed using a 1,300-scenario validation workload and public benchmark validation using HotpotQA and RAGTruth. In the 1,300-scenario evaluation, the proposed method achieved 91.31% routing accuracy, 85.00% grounded-answer rate, 85.38% unsupported-answer detection, and 79.85% fallback correctness, while reducing mean latency compared with always-on Static RAG. Paired statistical testing showed significant improvements over LLM-only, Static RAG, and Threshold RAG baselines for routing, grounding, unsupported-answer detection, fallback behavior, and latency. The paper also discusses deployment considerations including query-memory growth, stale no-retrieval labels, knowledge-base version tracking, privacy-preserving redaction, and instructor oversight for responsible educational use.

A RAG-Enhanced Human-in-the-Loop Framework for Automated Assessment of Engineering Laboratory Reports

Recent developments in Large Language Models (LLMs) have created new opportunities to automate educational assessment and reduce workload for instructors. However, concerns regarding grading consistency, transparency, and pedagogical reliability continue to limit the adoption of fully automated assessment systems. In this study, we propose a Human-in-the-Loop framework for the automated evaluation of engineering laboratory reports based on Retrieval-Augmented Generation (RAG). The proposed framework integrates text extraction, structured content extraction, contextual retrieval from grading rubrics and laboratory resources, rubric-based evaluation, automatic feedback generation, and instructor validation into a unified grading workflow. The RAG module retrieves contextual information so that the language model can generate assessments that align with the course goals and are based on educational information about the specific assignments, thus ensuring consistent evaluation based on the rubric. The system was tested using a set of 56 laboratory reports collected from undergraduate courses in Electrical and Electronics Engineering. The experimental results indicate a strong agreement between the grades provided by the AI and the instructor, with a Pearson Correlation Coefficient of 0.988, a Mean Absolute Error (MAE) of 3.55, and a Root Mean Squared Error (RMSE) of 3.77. Besides, 89.29% of the reports were scored within ±5 points of the instructor scores. The grading time was reduced from 392 minutes to 84 minutes, a workload reduction of 78.57%. The results demonstrate that the integration of Retrieval-Augmented Generation, rubric-based evaluation, and Human-in-the-Loop validation constitutes an effective approach for AI-supported assessment of engineering laboratory reports, maintaining instructor oversight and educational integrity.

GPT-only vs. GPT with RAG: A Study on Accuracy in Handling University-Specific Queries

While Large Language Models (LLMs) have demonstrated impressive capabilities in general natural language processing, their accuracy often diminishes in domain-specific contexts where precise, factual responses are crucial. This study addresses this limitation within the higher education sector by comparing two approaches to handling university-specific queries. We evaluate a Generative Pre-trained Transformers (GPT)-only model that relies on prompt engineering against a Retrieval-Augmented Generation (RAG) model that incorporates external university documents, specifically program flyers and a module handbook, integrated using Langchain. We benchmark both systems using 90 academic queries categorized by the question difficulty and assess their performance through automatic metrics and blind expert ratings. Our results demonstrate that RAG significantly outperforms the GPT-only approach, particularly for complex questions concerning curriculum and program structure. This research offers valuable insights for higher education institutions seeking to implement reliable and effective AI-powered solutions for student support and information provision.

Cheating with AI: Are Large Language Models Undermining Academic Integrity?

Domain adaptation of large language models for additive manufacturing using retrieval-augmented generation and fine-tuning

General-purpose large language models (LLMs) often struggle to generate reliable responses in specialized engineering domains due to limited domain grounding and insufficient exposure to structured technical knowledge. This study investigates practical strategies for adapting a foundation LLM to the additive manufacturing (AM) domain in order to improve answer accuracy, relevance and usability for expert-level question answering. AM knowledge is distributed across heterogeneous sources such as academic literature, manufacturer documentation, technical standards and procedural guides. Although general LLMs demonstrate strong linguistic capabilities, they frequently fail to retrieve and contextualize such domain-specific information. Two common approaches to address this limitation are domain-specific fine-tuning and retrieval-augmented generation (RAG). A curated AM corpus is constructed and used to evaluate three configurations based on LLaMA-3-8B: (1) a pretrained baseline model, (2) a RAG system that retrieves relevant document chunks from a vector database and (3) a model fine-tuned on raw domain text using quantized low-rank adaptation (QLoRA). Performance is evaluated using 200 expert-designed AM questions assessed by mechanical engineering experts for accuracy, relevance and overall preference. The RAG model consistently outperforms the baseline. Among the 200 questions, 75.5% of RAG responses are judged more accurate, 85.2% are preferred overall and 90.8% are rated more relevant than baseline responses. In contrast, fine-tuning on raw AM text reduces performance, producing more accurate answers in only 5.6% of cases and more relevant answers in 32.5% of cases. These results indicate that retrieval-augmented approaches provide a more effective pathway for adapting LLMs to specialized engineering domains than naive fine-tuning on unstructured technical data. RAG grounds responses in retrieved source documents at inference time, enabling higher accuracy and relevance without requiring model retraining.

Knowledge Distillation for Automated AI Tutor Evaluation

FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors, is introduced, which assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability.