NVIDIA announced that it will soon release “Nemotron-pretraining-Indic-v1,” a dataset of about 2.8 trillion tokens for AI pretraining, jointly developed with the Indian Institute of Technology Madras (IIT Madras) and its AI4Bharat group. The dataset covers the 22 languages specified in the Indian Constitution. NVIDIA also plans to release the GPU-enabled pipeline used to generate the data. The approach is designed to combine synthetic data based on English-language materials with natural text written locally.
A massive corpus covering 22 Indian languages in 34 configurations
The dataset covers the 22 official languages listed in the Indian Constitution. They include Hindi, Bengali, Tamil, Telugu and Urdu, and are the first languages of about 97% of India’s population. According to the 2011 census, that amounts to about 1.2 billion people.
Compared with English, Indian languages lack data for AI. This is an issue Mitesh Khapra, who leads AI4Bharat, has long highlighted, and the new dataset is positioned as one step toward closing that gap.
Let’s take a look at what it contains. All 22 languages have data written in their standard scripts. For 12 of them, the dataset also includes transliterations in Latin script—that is, the alphabet—bringing the total to 34 language-and-script combinations. The standard-script data accounts for about 2.601 trillion tokens, while the Latin-script versions account for about 224.6 billion. Both figures were counted using the Gemma-4 tokenizer.
One important caveat: these figures do not represent the number of “unique English documents.” Translations and versions written in different scripts, generated from the same sources, are counted separately. The Latin-script versions are not translations into English; they are Indian languages written in a romanized form. They can also serve as material for teaching models the same content in a different script.
Given how many people type Hindi in Latin script in emails and on social media, this design is practical. For Japanese, it might be like training on both text written in hiragana and kanji and text entered using romaji.
A four-stage generation process that goes beyond translation
The source material comes from 15 subsets of Nemotron-CC-v2, including web text, question-and-answer content, math and code. The material is processed using NVIDIA’s open-source synthetic data generation tool, “NeMo Data Designer,” which handles everything from translation to quality evaluation and revision.
The output retains identifiers for the source documents, making it possible to trace which documents the data came from. The ease of auditing the data later is one of the strengths of open data.
For 12 languages, the process has four stages: translating English documents, evaluating quality, revising low-rated output, and converting it to Latin script. Translation is done with Gemma-4-31B-it, which translates entire documents at once. This is intended to preserve context across paragraphs, tables and equations. It is also worth noting that the prompts instruct the model to leave code, equations and URLs untranslated.
The same model evaluates the translations against the original text, using criteria that cover meaning, fluency, terminology, style and formatting, based on MQM, a framework for classifying translation errors. Outputs scoring 75 or higher out of 100 are accepted. Lower-scoring outputs are revised using the evaluation feedback and scored again. This revision process salvaged 438.4 billion tokens across the standard- and Latin-script versions, and that amount is already included in the total.
The Latin-script versions are also produced through a carefully designed process. Rather than romanizing directly from English, the system converts the accepted standard-script translation, then checks the result against the original translation. An initial human comparison found that this two-step approach resulted in less fabricated content and fewer remnants of English.
Dedicated translation models cover the remaining 10 languages
The 10 languages without Latin-script versions are Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Meitei, Sanskrit, Santali and Sindhi. Translation is done using “Indic Translate,” a translation model from Bodhan AI, a nonprofit organization founded at IIT Madras.
Gemma-4-31B-it handles evaluation, with the same acceptance threshold of 75 points. However, the outputs are not rewritten based on feedback, and only standard-script data is produced.
The process also includes language-specific adjustments. Long Santali documents are translated in overlapping windows and then reassembled. For Meitei, the text is first translated into Bengali script and then converted into the language’s own Meitei Mayek script. These measures reflect the distinctive challenges posed by the wide variation in writing systems across India’s languages.
How much can we infer about performance gains?
One indication of the dataset’s impact comes from continued pretraining of Nemotron 3.5 Super VL. At this stage, 10% of the 100-billion-token training budget—10 billion tokens—was allocated to a mix of some of the 12 languages in the new corpus and natural-language text from India. The remaining 90% retained the proportions of the earlier training mix.
As a result, scores rose across all five benchmarks—MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K—by 4.98 to 9.82 points. However, this comparison looks at checkpoints before and after continued pretraining. It does not isolate the contribution of synthetic data alone. What is clear is that the blend of synthetic data and natural text worked.
Looking at the completed model as a whole, scores improved on five of six task evaluations measuring knowledge, reasoning, math, coding, and translation, compared with the previous-generation Nemotron 3 Super. Meanwhile, IFEval 1.5, which measures instruction following, fell by 2.04 points.
On Sarvam AI's IndiVibe evaluation, which measures conversational quality in Indian languages, all four criteria improved: helpfulness, fluency, fidelity to language and script, and appropriate response length.
Making the data usable, not just public
What stands out about this release is that it opens up not just the data itself, but also how it is produced. The translation plugin “Nemotron Bharat” for NeMo Data Designer includes prompts for translation and evaluation, rewriting functionality, document windowing, and transliteration support. This allows companies and research institutions with their own documents to create data for other domains using the same process.
The guidance on how to use it is also specific: first, test it on a small set of in-house documents and check whether code, equations, and formatting are preserved. Then scale up after review by speakers of the relevant languages. The plugin returns candidates and scores, so each team decides which records to include in training. Data can be loaded using Hugging Face's streaming feature, allowing users to inspect it without downloading the entire dataset.
Translated data cannot replace text written locally. Naturally written text carries regional knowledge and cultural context. This corpus is designed to be used alongside such natural text.
Quality evaluation is largely automated, with human checks focused mainly on samples. In the 12-language workflow, the model used for translation also serves as the evaluator, so the 10-language side requires broader verification by native speakers. Deduplication by source ID removes processing duplicates; it does not identify and consolidate semantically similar content.
With support from NVIDIA, AI4Bharat is also developing the evaluation benchmark suite Indic-LLM-Eval-Suite. Efforts to expand the data and to establish metrics for evaluating it are advancing in parallel. Can the foundations of AI, long centered on English, be strengthened through language-specific evaluation and a blend with natural text? This corpus for India's 22 languages could be a test case.
Comments