PhD Defense: Using CNNs to Understand Lighting without Real Labeled Training Data

The task of computer vision is to make computers understand the physical word through images. Lighting is the medium through which we capture images of the physical world. Without lighting, there is no image, and different lighting leads to different images of the same physical world. In this dissertation, we study how to understand lighting from images. Our biggest challenge is the lack of labeled data. We address this by showing how to effectively use labeled synthetic data.First, we propose Label Denoising Adversarial Network (LDAN) to estimate lighting from faces; for this task it is extremely difficult to collect ground truth labels. LDAN takes advantage of synthetic data to significantly boost performance. Second, we present a deep learning based portrait relighting algorithm. We generate the first large scale, high resolution, “in the wild” synthetic dataset. Our model trained on the proposed dataset outperforms existing methods significantly. Third, we study how to decompose an RGB image of a natural scene into reflectance, normal and lighting. Lighting of natural scenes is complex and not well studied. We propose a novel representation of this lighting using global and local Spherical Harmonics model to capture the lighting of a natural scene. A large scale synthetic dataset together with a weakly labeled real dataset are used to help train a network that achieves the state-of-the-art results.
Examining Committee:

Chair: Dr. David W. Jacobs Dean's rep: Dr. Rama Chellappa Members: Dr. Larry S. Davis Dr. Yaser Yacoob Dr. Tom Goldstein