Physics-Based Inverse Rendering

Abstract

This dissertation explores the challenges and solutions in physically-based inverse rendering, aiming to reconstruct detailed 3D scenes—including geometry, material properties, and lighting—from ordinary 2D images. The research journey began with the goal of making differentiable rendering more broadly applicable, particularly in scenarios where existing methods struggled with complex shapes, difficult initializations, or intricate material and lighting effects. The first step involved bringing differentiability to time-gated rendering, which is essential for advanced tasks such as Non-Line-of-Sight (NLOS) reconstruction that rely on the timing of light as it moves through a scene. To address the need for more flexible and robust geometric representations, a new inverse rendering pipeline was developed that combines the strengths of both implicit and explicit geometric representations. To accelerate initialization, neural rendering techniques (such as NeRF and NeuS) were incorporated to provide strong initial estimates for geometry, materials, and lighting, which are then refined using a physics-based differentiable renderer capable of handling complex global illumination. Finally, EnvMAP++ was introduced as a lightweight, parallax-aware lighting model, and Neural Implicit Evolution (NIE) was integrated for robust, topology-adaptive shape optimization, faithfully capturing the complex interplay between shape, glossy materials, and detailed local lighting. Collectively, these projects form a cohesive set of tools and methods that make physically-based inverse rendering more accurate, flexible, and reliable, helping to advance the field of computational scene understanding from images.

Guangyan Cai

Member of Technical Staff

I am interested in physics-based differentiable rendering and its applications, such as inverse rendering.