3d-photo-inpainting

[CVPR 2020] 3D Photography using Context-aware Layered Depth Inpainting

Github星跟踪图

[CVPR 2020] 3D Photography using Context-aware Layered Depth Inpainting

Open 3DPhotoInpainting in Colab

[Paper] [Project Website] [Google Colab]

We propose a method for converting a single RGB-D input image into a 3D photo, i.e., a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts when compared with the state-of-the-arts.

3D Photography using Context-aware Layered Depth Inpainting

Meng-Li Shih,
Shih-Yang Su,
Johannes Kopf, and
Jia-Bin Huang

In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.

Prerequisites

  • Linux (tested on Ubuntu 18.04.4 LTS)
  • Anaconda
  • Python 3.7 (tested on 3.7.4)
  • PyTorch 1.4.0 (tested on 1.4.0 for execution)

and the Python dependencies listed in requirements.txt

  • To get started, please run the following commands:
    conda create -n 3DP python=3.7 anaconda
    conda activate 3DP
    pip install -r requirements.txt
    conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit==10.1.243 -c pytorch
    
  • Next, please download the model weight using the following command:
    chmod +x download.sh
    ./download.sh
    

Quick start

Please follow the instructions in this section.
This should allow to execute our results.
For more detailed instructions, please refer to DOCUMENTATION.md.

Execute

  1. Put .jpg files (e.g., test.jpg) into the image folder.
    • E.g., image/moon.jpg
  2. Run the following command
    python main.py --config argument.yml
    
    • Note: The 3D photo generation process usually takes about 2-3 minutes depending on the available computing resources.
  3. The results are stored in the following directories:
    • Corresponding depth map estimated by MiDaS
      • E.g. depth/moon.npy, depth/moon.png
      • User could edit depth/moon.png manually.
        • Remember to set the following two flags as listed below if user wants to use manually edited depth/moon.png as input for 3D Photo.
          • depth_format: '.png'
          • require_midas: False
    • Inpainted 3D mesh (Optional: User need to switch on the flag save_ply)
      • E.g. mesh/moon.ply
    • Rendered videos with zoom-in motion
      • E.g. video/moon_zoom-in.mp4
    • Rendered videos with swing motion
      • E.g. video/moon_swing.mp4
    • Rendered videos with circle motion
      • E.g. video/moon_circle.mp4
    • Rendered videos with dolly zoom-in effect
      • E.g. video/moon_dolly-zoom-in.mp4
      • Note: We assume that the object of focus is located at the center of the image.
  4. (Optional) If you want to change the default configuration. Please read DOCUMENTATION.md and modified argument.yml.

License

This work is licensed under MIT License. See LICENSE for details.

If you find our code/models useful, please consider citing our paper:

@inproceedings{Shih3DP20,
  author = {Shih, Meng-Li and Su, Shih-Yang and Kopf, Johannes and Huang, Jia-Bin},
  title = {3D Photography using Context-aware Layered Depth Inpainting},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2020}
}

Acknowledgments

主要指标

概览
名称与所有者vt-vl-lab/3d-photo-inpainting
主编程语言Python
编程语言Python (语言数: 2)
平台
许可证Other
所有者活动
创建于2020-04-08 15:31:45
推送于2024-08-30 23:48:56
最后一次提交2021-06-21 18:14:14
发布数0
用户参与
星数7k
关注者数144
派生数1.1k
提交数89
已启用问题?
问题数191
打开的问题数95
拉请求数2
打开的拉请求数13
关闭的拉请求数10
项目设置
已启用Wiki?
已存档?
是复刻?
已锁定?
是镜像?
是私有?