Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language

Abstract

While Large Language Models (LLMs) demonstrate remarkable NLP capabilities, their performance on low-resource regional languages like Toba Batak remains underexplored. This study presents a rigorous empirical evaluation of four LLMs—ChatGPT, Gemini, Perplexity, and the domain-adapted Toba-LLM—on prompts written in the Toba Batak language using the CRISP-DM framework. Using a benchmark dataset spanning five task domains (information-seeking, instruction-following, creative generation, role-playing, and text-based tasks), model outputs were evaluated across six metrics: lexical similarity, keyword overlap, length ratio, multilingual cosine similarity, cultural relevance, and fluency. Experimental results reveal that general-purpose models outperform Toba-LLM in semantic similarity (cosine similarity: ChatGPT 0.628, Gemini 0.599) and fluency (ChatGPT 0.968, Gemini 0.945), whereas Toba-LLM demonstrates superior lexical alignment (0.0597) and conciseness (length ratio of 2.895). Nevertheless, cultural relevance remains a major challenge across all models (scores between 0.326 and 0.559), with qualitative findings highlighting frequent language inconsistency and Indonesian dominance bias. These findings emphasize the necessity of culturally grounded metrics, enriched regional corpora, and language-consistency constraints to advance LLM applicability in digital humanities and regional language preservation.