Zero-Shot Evaluation of Multilingual Models for Bidirectional English–Indic Machine Translation

Abstract

The recent outbreak of Large Language Models~(LLMs) has radically changed the state of machine translation~(MT), which has provided fresh possibilities to low-resource and morphologically rich language pairs. The paper is a detailed comparative analysis of five state-of-the-art models, BLOOMZ, IndicTrans2, Krutrim, Sarvam~AI, and Aya-101, in a zero-shot translation paradigm between English and four typologically different languages, Indic languages: Gujarati, Malayalam, Hindi, and Tamil. The analysis compares general-purpose multilingual LLMs to specialised neural machine translation systems (NMT) in a systematic way balancing the generality of multilingual coverage and task specialisation. Every model is tested on the FLORES-200 DevTest corpus against three complementary testing metrics, BLEU, chrF++ and COMET. The languages selected represent two different linguistic families, that is, Indo-Aryan (Hindi, Gujarati) and Dravidian (Tamil, Malayalam), which gives them a principled ground of cross-family comparison. Experimentation findings indicate that there is significant performance difference among language pairs, translation directions and models. Specialised Indic models are proven to have marked morphologically rich languages of Dravidian language families on both semantic and character-level measures, LLM instruction-tuned models are also shown to achieve good performance in limited resources environments with respect to BLEU. The results of our work provide practical recommendations in the selection of LLM in the production Indic MT pipeline and emphasize the importance of the linguistic family factor in the assessment of the multilingual model.