Neural-PBIR Reconstruction of Shape, Material, and Illumination

Oct 1, 2023·
Cheng Sun
Cheng Sun
Guangyan Cai
Guangyan Cai
Kai Yan
Kai Yan
Cheng Zhang
Cheng Zhang
Carl Marshall
Carl Marshall
Jia-Bin Huang
Jia-Bin Huang
Shuang Zhao
Shuang Zhao
Zhao Dong
Zhao Dong
· 0 min read
Abstract
Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination based on 2D images (e.g., photographs) of the object has been a long-standing problem in computer vision and graphics. In this paper, we introduce an accurate and highly efficient object reconstruction pipeline combining neural based object reconstruction and physics-based inverse rendering (PBIR). Our pipeline firstly leverages a neural SDF based shape reconstruction to produce high-quality but potentially imperfect object shape. Then, we introduce a neural material and lighting distillation stage to achieve high-quality predictions for material and illumination. In the last stage, initialized by the neural predictions, we perform PBIR to refine the initial results and obtain the final high-quality reconstruction of object shape, material, and illumination. Experimental results demonstrate our pipeline significantly outperforms existing methods quality-wise and performance-wise. Code:
Type
Publication
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Cheng Sun
Authors
Cheng Sun
Research Scientist at Nvidia
Guangyan Cai
Authors
Ph.D. Candidate
Kai Yan
Authors
Kai Yan
Ph.D. Candidate in Computer Science
Cheng Zhang
Authors
Cheng Zhang
Research Scientist at Meta Reality Labs
Carl Marshall
Authors
Carl Marshall
Research Scientist at Meta Reality Labs
Jia-Bin Huang
Authors
Jia-Bin Huang
Associate Professor of Computer Science at University of Maryland, College Park
Shuang Zhao
Authors
Shuang Zhao
Assistant Professor of Computer Science at the UC Irvine
Zhao Dong
Authors
Zhao Dong
Research Scientist at Meta Reality Labs