Low-resource hallucination detection in LLMs on multi-task datasets via iterative pseudo-labeling using confidence thresholding and active learning

Abstract

Large Language Models have advanced natural language generation, but they often produce outputs that are grammatically correct yet factually incorrect or misleading. This issue, commonly known as hallucination, reduces the reliability of such systems, especially in domains such as law, medicine, journalism, and education, where factual accuracy is important. This work presents a systematic low-resource framework for hallucination detection across three natural language processing tasks, including machine translation, definition modeling, and paraphrase generation. The experiments are conducted on the SemEval 2024 Task 6 SHROOM dataset, which consists of a small labeled set and a larger set of unlabeled data across three text generation tasks. An iterative self-training approach based on transformer models is employed, where a small manually labeled set is expanded using confidence-based pseudo-labeling. Active learning, ensemble methods and a Query by Committee strategy are used to guide sample selection and improve stability under limited supervision. The models are evaluated using accuracy and Spearman correlation to capture classification performance and prediction consistency. The proposed setup achieves an accuracy of 0.78 and a Spearman correlation of 0.6626. In addition, QLoRA-based large language models are evaluated as lightweight baselines for comparison with the proposed framework. The work presents a lightweight, semi-supervised, and reproducible framework that achieves competitive performance without relying on resource-intensive models. It serves as a practical baseline, highlights key challenges, and provides a foundation for more advanced hallucination detection systems in real-world applications.