Skip to content
Review

Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

Oct 2026 · 0 citations · 88 references
Computer Science

TL;DR

This work introduces Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form and introduces the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation.

Abstract

Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based filtering has proven effective in selecting high-quality subsets from web-scale corpora, especially for high-resource languages, low-resource languages face challenges d...

Vinko Sabolcec, Bettina Messmer, Yassine Turki et al. · 0 citations

Data-Efficient Adaptation of Multilingual LLMs to Ukrainian

This work combines vocabulary surgery for tokenizer adaptation without full retraining, cross-lingual transfer of quality classifiers via translation, enabling filtering without target-language annotations, and generation of instruction data through translation, task conversion, and targeted synthesis.

Yurii Paniv, Bohdan Didenko, Mykola Haltiuk et al. · 4 citations
#machine learning Preprint Oct 2026

AdaptTok-Indic: Script-Aware Adaptive Tokenization for Indic Languages

Results suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance, and suggest that Indic-specialized adaptive segmentation can reduce tokenization disparity while maintaining competitive downstream performance.

Binayak Mukherjeea, Moitreya Nandi, Arindam Panja · 0 citations

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

COILD, an Indic-centric parallel corpus comprising over 1.16 million human-translated and human-verified sentence pairs, covers 20 Indian language pairs across the Indo-Aryan, Dravidian, Tibeto-Burman, and Austro-Asiatic language families, and provides a valuable training and evaluation resource for advancing multiling...

Kshetrimayum Boynao Singh, Nitin Kumar Mishra, P. Dutta et al. · 0 citations
#artificial intelligence Preprint Oct 2026

How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

Safety alignment for Large Language Models is heavily focused on English with limited multilingual representation, particularly damaging low-resource language performance. Related open training datasets are often limited in size or do not follow region-localized harmful topics to broaden the scope of alignment beyond c...

Max Neuwinger, H. Yukhymenko, Imanol Schlag · 0 citations
Conference Open access 2026

MTG-LIRA: Scaling Neural Machine Translation for Low-Resource Indian Languages through Multilingual Learning and Synthetic Data

Low-resource Indian languages continue to face significant challenges in neural machine translation due to the scarcity of high-quality parallel corpora and the diversity of scripts and linguistic characteristics. This paper presents MTG-LIRA’s submission to the WMT 2026 Low-Resource Indic Language Translation Shared T...

Vedarth Hede, Neha Gupta, H. Bapat et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.