Neural-PBIR Reconstruction of Shape, Material, and Illumination
Oct 1, 2023··
0 min read
Cheng Sun
Guangyan Cai
Kai Yan
Cheng Zhang
Carl Marshall
Jia-Bin Huang
Shuang Zhao
Zhao Dong
Abstract
Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination based on 2D images (e.g., photographs) of the object has been a long-standing problem in computer vision and graphics. In this paper, we introduce an accurate and highly efficient object reconstruction pipeline combining neural based object reconstruction and physics-based inverse rendering (PBIR). Our pipeline firstly leverages a neural SDF based shape reconstruction to produce high-quality but potentially imperfect object shape. Then, we introduce a neural material and lighting distillation stage to achieve high-quality predictions for material and illumination. In the last stage, initialized by the neural predictions, we perform PBIR to refine the initial results and obtain the final high-quality reconstruction of object shape, material, and illumination. Experimental results demonstrate our pipeline significantly outperforms existing methods quality-wise and performance-wise. Code:
Type
Publication
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Authors
Cheng Sun
Research Scientist at Nvidia
Authors
Kai Yan
Ph.D. Candidate in Computer Science
Authors
Cheng Zhang
Research Scientist at Meta Reality Labs
Authors
Carl Marshall
Research Scientist at Meta Reality Labs
Authors
Jia-Bin Huang
Associate Professor of Computer Science at University of Maryland, College Park
Authors
Shuang Zhao
Assistant Professor of Computer Science at the UC Irvine
Authors
Zhao Dong
Research Scientist at Meta Reality Labs