💼 About Me

Xuehai He is a Ph.D. in Computer Science at the University of California, Santa Cruz working with Xin Eric Wang. His research work mainly revolves around Multimodal Learning and Machine Learning. Previously, he was at UC San Diego working with Pengtao Xie. He began his research at the University of Electronic Science and Technology of China.

📰 News

🛠️ 2025.02: Co-organize the CVPR 2025 Workshop – Computer Vision in the Wild.🎉 Host our MMWorld benchmark there.🔥
🏆 2025.02: One paper accepted to CVPR 2025.
🥇 2025.01: Two papers accepted to ICLR 2025.🚀
🥈 2024.08: One paper accepted to TMLR 2024.
🧑‍💻 2024.01: Rejoin Microsoft as a research intern.

📝 Selected Publications

The symbol * indicates equal contribution

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

Xuehai He, Weixi Feng*, Kaizhi Zheng*, Yujie Lu*, Wanrong Zhu*, Jiachen Li*, Yue Fan*, Jianfeng Wang, Linjie Li, Zhengyuan Yang, Kevin Lin, William Yang Wang, Lijuan Wang, Xin Eric Wang.
ICLR, 2025. [Project Website]

Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners

Xuehai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Eric Wang.
TMLR, 2024. [Project Website]

Parameter-efficient Model Adaptation for Vision Transformers

Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, Xin Eric Wang.
AAAI, 2023. [Project Website]

📝 Selected Preprints

Mojito: Motion Trajectory and Intensity Control for Video Generation

Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang.
[PDF]

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens

Kaizhi Zheng*, Xuehai He*, Xin Eric Wang.
[PDF]

VLM4D: Towards Spatiotemporal Awareness in Vision Language Models

Shijie Zhou*, Alexander Vilesov*, Xuehai He*, Ziyu Wan, Shuwang Zhang, Aditya Nagachandra, Di Chang, Dongdong Chen, Xin Eric Wang, Achuta Kadambi.
[PDF]

🔍 Publications

See Google Scholar for fully updated one.

Click to expand publications

Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, William Yang Wang. Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning. ICML, 2024.

Kenan Jiang*, Xuehai He*, Ruize Xu, Xin Eric Wang. ComCLIP: Training-Free Compositional Image and Text Matching. NAACL, 2024.

Kaizhi Zheng, Xiaotong Chen, Xuehai He, Jing Gu, Linjie Li, Zhengyuan Yang, Kevin Lin, Jianfeng Wang, Lijuan Wang, Xin Eric Wang. EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing. ICLR, 2025.

Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, Kevin Lin, William Yang Wang, Lijuan Wang, Xin Eric Wang. MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos. ICLR, 2025.

Xuehai He, Jian Zheng, Jacob Zhiyuan Fang, Robinson Piramuthu, Mohit Bansal, Vicente Ordonez, Gunnar A Sigurdsson, Nanyun Peng, Xin Eric Wang. FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation. TMLR, 2024.

Pengtao Xie, Xingchen Zhao, Xuehai He. Simultaneous Selection and Adaptation of Source Data via Four-Level Optimization. TACL, 2024.

Xuehai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Eric Wang. Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners. TMLR, 2024.

Weixi Feng*, Wanrong Zhu*, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Xuehai He, Sugato Basu, Xin Eric Wang, William Yang Wang. LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. NeurIPS, 2023.

Pengtao Xie, Xingchen Zhao, Xuehai He. Improve the Performance of CT-based Pneumonia Classification via Source Data Reweighting. Nature Scientific Reports.

Xuehai He, Xin Eric Wang. Multimodal Graph Transformer for Multimodal Question Answering. EACL, 2023.

Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, William Yang Wang. Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. ICLR, 2023.

Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, Xin Eric Wang. Parameter-efficient Model Adaptation for Vision Transformers. AAAI, 2023.

Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Eric Wang. CPL: Counterfactual Prompt Learning for Vision and Language Models. EMNLP, 2022.

Tarun Gupta, Xuehai He, Mostofa Rafid Uddin, Xiangrui Zeng, Andrew Zhou, Jing Zhang, Zachary Freyberg, Min Xu. Self-supervised learning for macromolecular structure classification based on cryo-electron tomograms. Frontiers in Physiology.

Xuehai He*, Zhuo Cai*, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, Pengtao Xie. Towards Visual Question Answering on Pathology Images. ACL, 2021.

Wenmian Yang, Guangtao Zeng, Bowen Tan, Zeqian Ju, Subrato Chakravorty, Xuehai He, Shu Chen, Xingyi Yang, Qingyang Wu, Zhou Yu, Eric Xing, Pengtao Xie. On the Generation of Medical Dialogues for COVID-19. ACL, 2021.

🚲 Service

🗒️ Conference Reviewer: ICASSP’19, IJCAI’21, AAAI’21, CVPR’21-‘24, ICCV’21-‘23, ECCV’22, NeurIPS’22-‘23, EMNLP’22-‘23, ACL’23-‘24, ICML’23-‘24.
✒️ Journal Reviewer:
- IEEE Access’19’20
- Transactions on Pattern Analysis and Machine Intelligence (TPAMI)’24
👥 Program Committee Member:
- NeurIPS 2021 Workshop: Self-Supervised Learning – Theory and Practice [Link]
🤝 Workshop Co-organizer:
- AAAI 2021: Trustworthy AI for Healthcare [Link]
- ECCV 2022: Workshop on Computer Vision in the Wild [Link]
- CVPR 2024: 4th Workshop on Computer Vision in the Wild
🔍 Workshop Reviewer:
- NeurIPS 2022: Efficient Natural Language and Speech Processing (ENLSP-II) [Link]
- ICML 2021: Self-Supervised Learning for Reasoning and Perception [Link]