NVIDIA Introduces 2.8T-Token Indian Pretraining Corps and Open Pipeline

NVIDIA, in collaboration with IIT Madras and its AI4Bharat lab, on October 16, 2026 introduced Nemotron-pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.8 trillion tokens in 22 Indian languages that will be released soon, along with the open, GPU-accelerated data generation pipeline used to create it.

The 22 languages are those listed in the Indian Constitution, which the announcement describes as the first languages for 97% of Indians, nearly 1.2 billion citizens according to the 2011 census. The authors of the post include AI4Bharat researchers Mohammed Safi Ur Rahman Khan, Mitesh Khapra and Anoop Kunchukuttan and Niket Agarwal, Raviraj Joshi, Anusha Kamath, Markus Kliegl, Yev Meyer, Sriharsha Niverty, Kiran Praveen and Utkarsh Vaidya of NVIDIA. Khapra, who heads AI4Bharat, identified the main problem behind Indian language technology following English: “we don’t have enough data for Indian languages.”

The pipeline captures the complete build recipe, including prompts, LLM workflows, and quality checks, so developers can adapt it to their source material. NVIDIA said that each record traces back to a curated source document, and each output is compared to its input.

Corpus composition and linguistic coverage

The corpus contains 2,601.0 billion native script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. NVIDIA notes that the counts describe the retained corpus, including accepted rewrites, rather than the token budget of a training cycle, and that the counts depend on the tokenizer used.

Across 34 linguistic script configurations, the corpus provides native script data for all 22 languages and romanized text for twelve. Twelve languages have both native and romanized script data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu. Ten have data only in native script: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali and Sindhi.

The post defines native script as the writing system, with the text still translated from English, while romanized means Indian language text written in Latin script. For each of the twelve dual-script languages, romanization covers a subset of the accepted native script translations.

Two translation workflows

Generation begins with documents in 15 Nemotron-CC-v2 subsets covering web text, question-and-answer material, math, code, and other specialized content. NeMo Data Designer, NVIDIA’s open-source synthetic data framework, performs translation, quality checks, and targeted repairs, maintaining source identifiers so output can be traced back to the original document.

For the twelve dual-script languages, the documents are translated in full using Google’s Gemma-4-31B-it, which the team said it selected by comparing patterns on the full documents. The instructions distinguish the prose to be translated from the content to be preserved, including code, equations, URLs, and formatting. The same model then evaluates each output against its English source using a rubric inspired by the Multidimensional Quality Metrics framework, covering meaning, fluency, terminology, style and formatting. Only results that obtain a minimum score of 75 out of 100 are retained; Lower scoring results are corrected by assessment feedback and re-evaluated against the same threshold. NVIDIA reports that this repair cycle recovered 438.4 billion tokens of native and romanized data, already included in the corpus total, and that sampled human reviews and tests with deliberately introduced errors informed improvements to requests and checks.

Accepted native script translations provide the starting point for romanized data through a separate script conversion step. According to NVIDIA, an initial human comparison found that this two-step approach produces fewer hallucinatory adjuncts and less residual English than generating romanized text directly from English.

For the other ten languages, the translation uses Indic Translate, a document-level model for English and 22 Indian languages from Bodhan AI, a non-profit center for artificial intelligence in education incubated at IIT Madras, with Gemma-4-31B-it evaluating each translation against its English source. Eight languages maintain the whole-document approach; longer Santali documents are translated into overlay windows and reconstructed, and Manipuri is translated into Bengali script before conversion to Meitei Mayek, its native script. These workflows maintain the threshold of 75 out of 100 and only produce data in native scripts, without rewriting based on feedback.

Continued pre-workout results

The Nemotron 3.5 Super VL technical report compares the checkpoint immediately before Indic continued pretraining with the checkpoint following that phase. The configured budget was 100 billion training tokens, with 10% (10 billion tokens) allocated to a mix of synthetic and natural Indian text and the remaining 90% maintaining the relative proportions of the previous training mix. Scores increased on all five reported Indian knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K, with gains ranging from 4.98 to 9.82 percentage points. NVIDIA notes that the comparison covers a training phase that combines synthetic and natural Indian data rather than isolating the contribution of the synthetic corpus, and that GSM8K excludes Odia because the evaluation exceeded the service context boundary.

Compared to the Nemotron 3 Super, the released Nemotron 3.5 Super VL scores higher in five of the six activity benchmarks shown, spanning knowledge, reasoning, math, coding, and translation, while subsequent instructions on IFEval 1.5 lost 2.04 points, according to the post. It also improves all four dimensions of IndiVibe, Sarvam AI’s open benchmark of assistant-style suggestions in Indian languages: usefulness, fluency, speech and writing fidelity, and appropriate verbosity.

NVIDIA said it plans to build on this corpus in future Nemotron models, and that evaluating the entire corpus, including the recovered rewrites and the ten-language pool, remains the next step for controlled studies.

Availability and limitations stated

Dataset configurations use language, script, and region tags, with hi-Deva-IN denoting Devanagari Hindi and hi-Latn-IN denoting Romanized Hindi, and Hugging Face dataset streaming allows readers to examine records without downloading the entire corpus. The Nemotron Bharat open source translation plugin for NeMo Data Designer provides translation suggestions and evaluation, optional feedback-based rewriting, document windows, and script remapping support.

NVIDIA cautions that translation does not replace text originally written in Indian languages or ensure coverage of local knowledge and culture, and recommends combining the corpus with appropriately sourced natural Indian text. The quality assessment is mainly automated, complemented by a sampled human review whose coverage varies from language to language, and in the twelve-language workflow the translator also acts as a judge. The post states that preserved text may still contain errors, that the ten-language workflow needs broader validation by native speakers, and that source ID checks remove duplicate processing without performing semantic deduplication of similar content.

Complementary benchmark suite

In addition to the corpus, AI4Bharat at IIT Madras, with support from NVIDIA, has introduced Indic-LLM-Eval-Suite, an open and evolving collection of offline benchmarks for evaluating LLMs in Indian languages.

The first release brings together existing Indian benchmarks, updated versions including MILU-1.5 and IndicIFEval-1.5, and adapted English benchmarks across eight ability categories: general chat, following instructions, mathematics, STEM, common sense reasoning, knowledge, coding and translation. Assessments are integrated with NVIDIA NeMo-Gym, providing a common assessment interface while maintaining benchmark-specific scoring procedures.

AI4Bharat reports that its audit of MILU found incomplete questions, missing figures, and incorrect translations, which were addressed in the revised MILU-1.5 version, and that it fixed issues in the IndicIFEval evaluation code where some constraints were incorrectly checked or silently ignored. The suite’s construction methodology combines benchmark-based translation with automatic quality assessment, targeted correction and native speaker verification.

AI4Bharat said it evaluated several contemporary open-weight Indian and multilingual LLMs within the suite and found that no single model consistently performs better across all tasks. The lab describes the initial release as a starting point and says it plans to add benchmarks, incorporate community contributions and periodically update results to track the progress of Indian LLMs over time.