InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

Zhiheng Liu1,4*, Hao Ouyang2,3*, Qiuyu Wang3, Ka Leong Cheng2,3, Jie Xiao1,4, Kai Zhu4, Nan Xue3, Yu Liu4, Yujun Shen3, Yang Cao1†
1USTC  2HKUST  3Ant Group  4Alibaba Group
Paper Code

Gaussian Inpainting


Point Cloud & Mesh



Pipeline



Top: To remove a target from the optimized 3D Gaussians, our Infusion first inpaints a selected one-view RGB image and applies the proposed diffusion model for depth inpainting to the depth projection of the targeted 3D Gaussians. The progressive scheme addresses view-dependent occlusion issues by utilizing other unobstructed viewpoints.

Bottom: A detailed view of the training pipeline for the depth inpainting U-Net is presented. We employ a mask-driven denoising diffusion for training of the U-Net, which utilizes a frozen latent tokenizer by taking the RGB image and depth map as inputs.




Texture Editing


InFusion allows users to modify the appearance and texture of targeted areas with ease.

Object Insertion & Completion


Infusion allows user to project objects into a real three-dimensional scene through editing a single image.

BibTeX

@article{liu2024infusion,
      title={InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior},
      author={Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, Yang Cao},
      journal={arXiv preprint arXiv:2404.11613},
      year={2024}
    }