Mohammed Safi Ur Rahman Khan

AI4Bharat Lab

Indian Institute of Technology, Madras

Chennai, Tamil Nadu, India

I am Safi (صفی), a third year PhD Student at the Wadhwani School of Data Science and AI (DSAI) @ IIT Madras & AI4Bharat Lab where I am advised by Prof. Mitesh M. Khapra. My research explores Resources and Evaluation methods for Multilingual Large ‘X’ Language Models (where X = ‘Text’ or ‘Vision’ or ‘Audio’). I’m fortunate to be a Google PhD Fellow (2025) and have received Outstanding Paper Awards at EMNLP 2024 and ACL 2024.

Alongside my PhD, I work as an LLM Researcher at Bodhan AI, where I explore how LLMs can transform education through intelligent tutoring systems, personalized learning experiences, agentic AI, and robust evaluation frameworks for real-world educational applications.

Earlier, I was also a Research Fellow at Sarvam, helping in building Sovereign AI for India 🇮🇳. Before that, I was an AI Resident at the AI4Bharat Lab at IIT Madras, where I was fortunate to be part of the IndicLLMSuite (guided by Prof. Mitesh M. Khapra). I did my M.Tech in Computer Science and Engineering before that from IIT Madras (again!!) where I got to work on “Narrow Domain Adaptation of Speech Recognition Systems” guided by Prof. Pratyush Kumar and (you guessed it!) Prof. Mitesh M. Khapra.

news

Jul 11, 2026	Another work on LLM meta-evaluation - Seeing Isn’t Believing (the multimodal successor of FBI) - has been accepted to COLM 2026!!
Jun 10, 2026	Our work on large scale preference based evaluation of TTS models (Preferences Of a Voice-First Nation) has been accepted to Interspeech 2026. Time to go to Sydney?
Oct 24, 2025	I’m honored to be selected as a recipient of the Google PhD Fellowship 2025 in the Natural Language Processing Track. الحمد لله. Grateful for the support and guidance of my mentors and collaborators throughout this journey.
Oct 21, 2025	I’ll be presenting a tutorial titled “Data and Model Centric Approaches for Expansion of Large Language Models to New Languages” @ EMNLP-2025 in Suzhou, China 🇨🇳. Please do attend if you are around. Reference material of the tutorial can be found here.
Jun 29, 2025	Will be attending ACL-2025 in Vienna 🇦🇹 Insha’Allah!! Will be co-presenting - FairITales, CIA, FERMAT, and BhasaAnuvaad. Please catch us up at our posters to know more about our work.
Jun 01, 2025	I’ll be joining Sarvam AI as a Research Fellow to help with the Sovereign LLM efforts. Will be working with the Alignment and Evaluations team to build an India-first 🇮🇳 model!!

latest posts

Nov 10, 2025	Indic LLM-Arena \| AI4Bharat Blog
Mar 14, 2024	Indic LLM Suite \| AI4Bharat Blog

selected publications

COLM 2026

Seeing Isn’t Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand, and Mitesh M. Khapra

COLM 2026, 2026

Bib PDF Code Data

@article{khan2026seeing,
  title = {Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models},
  author = {Khan, Mohammed Safi Ur Rahman and Suryanarayanan, Sanjay and Anand, Tushar and Khapra, Mitesh M.},
  year = {2026},
  journal = {COLM 2026},
  data = {https://huggingface.co/datasets/ai4bharat/Focus},
}

ACL 2025

FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes

Janki Atul Nawale^*, Mohammed Safi Ur Rahman Khan^*, Janani D, Mansi Gupta, Danish Pruthi, and Mitesh M. Khapra

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

Bib PDF Code Data

@article{nawale2025fairi,
  title = {FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes},
  author = {Nawale, Janki Atul and Khan, Mohammed Safi Ur Rahman and D, Janani and Gupta, Mansi and Pruthi, Danish and Khapra, Mitesh M.},
  year = {2025},
  journal = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  data = {https://huggingface.co/datasets/ai4bharat/Indic-Bias},
}

ACL 2025

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

Sumanth Doddapaneni^*, Mohammed Safi Ur Rahman Khan^*, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M. Khapra

Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

Bib PDF Code Data

@article{doddapaneni2024crosslingual,
  title = {Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs},
  author = {Doddapaneni, Sumanth and Khan, Mohammed Safi Ur Rahman and Venkatesh, Dilip and Dabre, Raj and Kunchukuttan, Anoop and Khapra, Mitesh M.},
  year = {2025},
  journal = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  data = {https://huggingface.co/collections/ai4bharat/cia-suite-66ea9a7e18a6c70bd8de27a1},
}

NAACL 2025

MILU: A Multi-task Indic Language Understanding Benchmark

Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen

Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025

Bib PDF Code Data

@article{verma2024milu,
  title = {MILU: A Multi-task Indic Language Understanding Benchmark},
  author = {Verma, Sshubam and Khan, Mohammed Safi Ur Rahman and Kumar, Vishwajeet and Murthy, Rudra and Sen, Jaydeep},
  year = {2025},
  journal = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics},
  data = {https://huggingface.co/datasets/ai4bharat/MILU},
}

EMNLP 2024

Finding Blind Spots in Evaluator LLMs with Interpretable Checklists

Sumanth Doddapaneni^*, Mohammed Safi Ur Rahman Khan^*, Sshubam Verma, and Mitesh M. Khapra

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

Outstanding Paper Bib PDF Code Data

EMNLP-2024 Outstanding Paper Award

@article{doddapaneni2024finding,
  title = {Finding Blind Spots in Evaluator LLMs with Interpretable Checklists},
  author = {Doddapaneni, Sumanth and Khan, Mohammed Safi Ur Rahman and Verma, Sshubam and Khapra, Mitesh M.},
  year = {2024},
  journal = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  month = nov,
  address = {Miami, Florida, USA},
  publisher = {Association for Computational Linguistics},
  pages = {16279--16309},
  data = {https://huggingface.co/datasets/ai4bharat/FBI},
}

ACL 2024

IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

Mohammed Safi Ur Rahman Khan^*, Priyam Mehta^*, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024

Outstanding Paper Bib PDF Code Data

ACL-2024 Outstanding Paper Award

@article{khan2024indicllmsuite,
  title = {IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages},
  author = {Khan, Mohammed Safi Ur Rahman and Mehta, Priyam and Sankar, Ananth and Kumaravelan, Umashankar and Doddapaneni, Sumanth and G, Suriyaprasaad and G, Varun Balan and Jain, Sparsh and Kunchukuttan, Anoop and Kumar, Pratyush and Dabre, Raj and Khapra, Mitesh M.},
  year = {2024},
  journal = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  month = aug,
  address = {Bangkok, Thailand},
  publisher = {Association for Computational Linguistics},
  pages = {15831--15879},
  data = {https://huggingface.co/collections/ai4bharat/indicllmsuite-65ee7d225c337fcfa0991707},
}