Mohammed Safi Ur Rahman Khan

PhD Scholar • DSAIAI4BharatIIT Madras

NewProfilepic.jpeg

AI4Bharat Lab

Indian Institute of Technology, Madras

Chennai, Tamil Nadu, India

I am Safi (صفی), a first year PhD Student at the Wadhwani School of Data Science and Artificial Intelligence (DSAI) @ IIT Madras & AI4Bharat Lab where I am advised by Prof. Mitesh M. Khapra. My current research focuses on “all” Data and Evaluation of Large Language Models.

Previously, I was an AI Resident at the AI4Bharat Lab at IIT Madras, where I was fortunate to be part of the IndicLLMSuite (guided by Prof. Mitesh M. Khapra). I did my M.Tech in Computer Science and Engineering before that from IIT Madras (again!!) where I got to work on “Narrow Domain Adaptation of Speech Recognition Systems” guided by Prof. Pratyush Kumar and (you guessed it!) Prof. Mitesh M. Khapra.

news

Nov 14, 2024 FBI wins 🏆 Outstanding paper too!! Alhamdulillah.
Nov 07, 2024 I’ll be in Miami 🇺🇸 for EMNLP 2024 to present FBI. Thank you Google for sponsoring this. Looking forward to connect with y’all!!
Nov 07, 2024 Finally something that was cooking for a while! Now people dont have to use translated MMLU because we release MILU.
Oct 19, 2024 We follow up FBI with CIA!! Check out our latest preprint on doing Cross Lingual Auto Evaluation here.
Sep 21, 2024 FBI has been accepted to EMNLP 2024. Miami here we come!!
Aug 14, 2024 IndicLLMSuite wins the Outstanding paper award 🏅 at ACL 2024!!

latest posts

selected publications

  1. arXiv
    MILU: A Multi-task Indic Language Understanding Benchmark
    Sshubam Verma, Mohammed Safi Ur Rahman Khan, Vishwajeet Kumar, Rudra Murthy, and Jaydeep Sen
    arXiv preprint arXiv: 2411.02538, 2024
  2. arXiv
    Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
    Sumanth Doddapaneni*Mohammed Safi Ur Rahman Khan*, Dilip Venkatesh, Raj Dabre, Anoop Kunchukuttan, and Mitesh M. Khapra
    arXiv preprint arXiv: 2410.13394, 2024
  3. EMNLP 2024
    EMNLP-2024 Outstanding Paper Award
    Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
    Sumanth Doddapaneni*Mohammed Safi Ur Rahman Khan*, Sshubam Verma, and Mitesh M. Khapra
    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024
  4. ACL 2024
    ACL-2024 Outstanding Paper Award
    IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
    Mohammed Safi Ur Rahman Khan*, Priyam Mehta*, Ananth Sankar, Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra
    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug 2024