Ismini Lourentzou

About

I am an Assistant Professor in the School of Information Sciences at the University of Illinois Urbana-Champaign, where I lead the Perception and LANguage (PLAN) Lab. My work spans multimodal machine learning, vision-language models, learning with limited supervision, and human-centric embodied AI.

Research

My research focuses on building adaptive, compositional AI agents that perceive, reason, and interact with the world. Key directions include:

• Multimodal learning and vision-language models.
• Spatial and temporal reasoning for grounded vision-language tasks.
• Human-agent interaction through demonstration, instruction following, and communication.
• AI for healthcare, science, and social good.

Selected Publications

For the full list, visit Google Scholar.

CVPR'25

CALICO: Part-Focused Semantic Co-Segmentation with Large Vision-Language Models

Kiet A Nguyen, Adheesh Juvekar, Tianjiao Yu, Muntasir Wahed, Ismini Lourentzou

Abstract

Recent advances in Large Vision-Language Models (LVLMs) have sparked significant progress in general-purpose vision tasks through visual instruction tuning. While some works have demonstrated the capability of LVLMs to generate segmentation masks that align phrases with natural language descriptions in a single image, they struggle with segmentation-grounded comparisons across multiple images, particularly at finer granularities such as object parts. In this paper, we introduce the new task of part-focused semantic co-segmentation, which seeks to identify and segment common and unique objects and parts across images. To address this task, we present CALICO, the first LVLM that can segment and reason over multiple masks across images, enabling object comparison based on their constituent parts. CALICO features two proposed components, a novel Correspondence Extraction Module, which captures semantic-rich information to identify part-level correspondences between objects, and a Correspondence Adaptation Module, which embeds this information into the LVLM to facilitate multi-image understanding in a parameter-efficient manner. To support training and evaluation, we curate MixedParts, a comprehensive multi-image segmentation dataset containing ∼2.4M samples across ∼44K images with diverse object and part categories. Experimental results show CALICO, finetuned on only 0.3% of its architecture, achieves robust performance in part-focused semantic co-segmentation.

PDF Project

WACV'25

ELBA: Learning by Asking for Embodied Visual Navigation and Task Completion

Ying Shen, Daniel Bis, Cynthia Lu, Ismini Lourentzou

Abstract PDF Project

Talks & Media

• April, 2024: VT Generative AI Applications in Social Science - Generative AI Trends.
• May 2024: Midwest Machine Learning Symposium (MMLS) - Commonsense for Video Localization.
• March, 2023: UMass CIIR Talk Series - Robust Visual Understanding w/ Limited Supervision.
• November, 2023: Curious Conversations Podcast Episode on AI Personal Assistants.

News

Teaching

I teach undergraduate and graduate courses on Machine Learning, Multimodal ML, and AI foundations. Recent courses include:

• IS 557: Machine Learning Team Projects (Spring 2025)
• IS 327: Concepts of Machine Learning (Fall 2024)
• CS6604@VT: Embodied Artificial Intelligence (Fall 2023)
• CS5824@VT: Advanced Machine Learning (Spring 2023)

Students

I recruit graduate students through the School of Information Sciences (iSchool), the Siebel School of Computing and Data Science (CS), and the Electrical and Computer Engineering (ECE) Department at UIUC. For more details, please check the Information for Prospective Students.