Pakistani Researcher Develops UrduBench to Evaluate AI Reasoning in Urdu
Muhammad Ali Shafique and his research team introduce a benchmark to assess how accurately AI models solve mathematical, scientific and reasoning problems in Urdu.

UrduBench Pakistani research introduces a new benchmark for testing AI reasoning in Urdu.
A Pakistani researcher at Kansas State University has helped develop a new benchmark to evaluate how well artificial intelligence (AI) models understand and solve problems in Urdu.
The project, called UrduBench, is led by Muhammad Ali Shafique. It aims to assess AI systems beyond their ability to generate fluent Urdu text. Instead, it tests their ability to solve mathematical problems, answer scientific questions and handle everyday reasoning tasks in the language.
Although popular AI systems can now communicate in Urdu, their fluency does not necessarily reflect their reasoning ability. UrduBench addresses this gap by providing researchers with a standardised way to measure AI performance in Urdu.
Four International Benchmarks Adapted for Urdu
Developing UrduBench involved more than translating questions from English. The researchers needed to preserve the meaning, context and structure of mathematical problems, scientific terminology and everyday situations.
To achieve this, the team used translations from multiple AI systems and incorporated human review. This approach helped prepare Urdu versions of four established evaluation datasets:
- MGSM: Multilingual mathematical reasoning problems.
- MATH-500: Challenging mathematical problems.
- CommonSenseQA: Questions testing everyday knowledge and common sense.
- OpenBookQA: Science questions requiring knowledge and reasoning.
Together, these datasets form UrduBench, allowing researchers to examine different aspects of AI reasoning in Urdu.
Testing Whether AI Models Can Reason in Urdu
The research also examines whether AI models maintain their reasoning ability across languages.
For example, a model might solve a mathematical problem correctly in English but struggle with the same question in Urdu. The team evaluated different AI models to investigate such performance differences.
SPELT Hosts 42nd International Conference to Promote Equitable Language Education
Researchers also examined language consistency during problem-solving. In particular, they investigated whether models continued reasoning in Urdu or switched to English.
The study found that multi-step and symbolic reasoning tasks remain challenging for AI models in Urdu. It also highlighted the importance of maintaining consistent language alignment for reliable multilingual reasoning.
The research, titled “UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop,” was published in the proceedings of the 2026 Workshop on Open Reasoning Across Cultures and Languages (ORACLE), under the Association for Computational Linguistics.
Supporting More Reliable AI for Urdu Speakers
UrduBench gives researchers and AI developers a practical framework to identify strengths and weaknesses in language-based AI systems. It can also help guide improvements in multilingual AI development.
As students, professionals and businesses increasingly rely on AI tools, accurate answers matter as much as natural communication. Users need systems that understand questions and solve problems correctly in their preferred languages.
Shafique’s research contributes to efforts to make AI more reliable and accessible for Urdu-speaking communities. It also offers a framework that researchers could adapt to evaluate reasoning in other languages with limited resources.
The work highlights an important challenge for the AI industry: making advanced reasoning capabilities available beyond English-speaking users.
