Nasscom Plans Benchmark for AI Models in Indian Languages
IT industry lobby is developing a national benchmarking framework to evaluate AI models across Indic languages, starting with a Hindi pilot and expanding to speech and multimodal tasks.
Topics
IT industry lobby Nasscom is building what could become a national benchmark for evaluating artificial intelligence models in Indian languages, a move aimed at creating common standards for developers and improving the credibility of homegrown systems in a crowded global market.
Nasscom, which represents more than 3,000 IT services firms and startups, said it has designed a methodology for the initiative after extensive consultations with enterprises, researchers, and developers.
The benchmark will test how well models perform across multiple Indic languages, addressing a gap that existing global platforms, which are built largely for English and European languages, have left unresolved.
“Benchmarks are the way to understand how robust AI systems are in real-world language use,” Nasscom AI head Ankit Bose said, adding that the framework is currently under peer review for publication in an international journal, and that Nasscom intends to approach the government to formalize the exercise as a national process.
The effort is timely. While Indian startups increasingly rely on open-source benchmarking platforms such as Hugging Face or evaluation suites such as MMLU, BIG-Bench, and HELM, these tools are often unsuitable for a multilingual context.
Developers have long complained that they do not reflect the diversity of local languages, accents, and the complex intermingling of English with vernacular speech that is common in the country.
The proposed Indian benchmark aims to fill that void. According to Bose, the initial rollout will focus on Hindi, before extending to other languages in phases.
The framework will evaluate models not only on text but also on speech and multimodal tasks, ensuring that the results reflect the diverse ways in which users interact with technology.
“As the initiative matures, we will expand to cover additional languages and modalities based on ecosystem readiness and research feedback,” he said.
Startups and developers are expected to be at the center of this initiative. Nasscom said it plans to host roundtables and consultations where participants can suggest metrics, test evaluation pipelines, and contribute real-world datasets.
The first such roundtable will take place on 7 October during the Nasscom Agentic AI Confluence in Bengaluru, where more than 30 stakeholders from across the ecosystem will gather.
As part of the open benchmarking exercises, startups will be offered access to datasets, tools, and leaderboards to test and improve their models, Bose said, adding that they will also be invited to co-author evaluation reports, contribute language data, and conduct pilot research linked to benchmark tasks.
Bose emphasized that this participatory model would ensure that the benchmarks are shaped by the community rather than imposed top-down.
“Through these avenues, startups and developers will not only shape the benchmark but also showcase their models against global standards, driving both credibility and visibility,” he said.
