Zhiheng Liu

I am a second-year Ph.D. student at the Department of Computer Science, The University of Hong Kong (HKU), advised by Prof. Ping Luo.

Before that, I obtained my master degree from University of Science and Technology of China (USTC), advised by Prof. Yang Cao.

My previous research primarily centered around generative models, with a focus on image, video, and 3D generation. Currently, my interests have shifted towards native multimodal models. I am also actively exploring long-context modeling, an area where I am currently an enthusiastic beginner.

I am always open to research discussions and collaborations; please feel free to contact me via email (zhihengl0528 AT connect.hku.hk).

Email  /  Google Scholar  /  Github

• [9. 2025] 3 papers accepted at NeurIPS 2025.
• [3. 2025] 3 papers accepted at CVPR 2025, including one Highlight.
• [12. 2024] We release DepthLab, a robust depth inpainting foundation model that can be applied to various downstream tasks to enhance performance.
• [12. 2024] We release The Matrix, a foundation world model for generating infinite-length, hyper-realistic videos with real-time, frame-level control.
• [7. 2024] LivePhoto accepted at ECCV 2024.
• [5. 2024] CCM accepted at ICML 2024.
• [4. 2024] We release InFusion for 3D inpainting via diffusion prior.
• [3. 2024] DreamVideo accepted at CVPR 2024.
• [1. 2024] DreamClean accepted at ICLR 2024.
• [12. 2023] This page is online. Discussions and collaborations are welcome.

(*: Equal contribution)

This work introduces Tuna, a native Unified Multimodal Model (UMM) that builds a continuous visual representation space for end-to-end image and video processing. Key highlights include:

• It achieves state-of-the-art results across a wide range of tasks, including image and video.
• The representation encoder plays a crucial role; using stronger pretrained encoders consistently and significantly improves performance across all multimodal tasks.
• Compared to previous models with decoupled encoders, Tuna's unified architecture avoids representation format mismatches, leading to superior results in both understanding and generation.

WorldWeaver is a framework for long-horizon video generation that unifies RGB and perceptual conditions, leveraging depth-guided memory and segmented noise scheduling to enhance structural and temporal consistency.

We propose a robust depth inpainting foundation model that can be applied to various downstream tasks to enhance performance.

We present an image-conditioned depth inpainting model, which uses the diffusion prior to inpaint 3D Gaussians and has very good geometric and texture consistency.

MangaNinja is a reference-based line art colorization method that enables precise matching and fine-grained interactive control.

We present LivePhoto, a real image animation method with text control. Different from previous works, LivePhoto truely listens to the text instructions and well preserves the object-ID.

Cones 2 uses a simple yet effective representation to register a subject. The storage space required for each subject is approximately 5 KB. Moreover, Cones 2 allows for the flexible composition of various subjects without any model tuning.

We explore the subject-specific concept neurons in a pre-trained text-to-image diffusion model. Concatenating multiple clusters of concept neurons representing different persons, objects, and backgrounds can flexibly generate all related concepts in a single image.

Design and source code from Jon Barron's website