PhD Defense: Scene and Action Understanding using Context and Knowledge Sharing

Complete scene understanding from video data involves spatio-temporal decision making over long sequences and utilization of world knowledge. We propose a method that captures edge connections between these spatio-temporal components or knowledge graphs through a graph convolution network (GCN). Our approach uses the GCN to fuse various information in the video like detected objects, human pose, scene information etc. for action segmentation. For certain functions like zero shot and few shot action recognition, we learn a classifier for unseen test classes through comparison with similar training classes. We provide the information about similarity features between the two classes through an explicit relationship map i.e. the knowledge graph. As an improvement over the previous technique, we study different kinds of knowledge graphs based on action phrases, verbs or nouns and demonstrate how they perform with respect to each other. We help build an integrated approach for zero/few-shot learning. We also show further improvements through adaptive learning of the input knowledge graphs and using a metric loss along with the L2 loss while training.To approach complete scene understanding from a different direction, we also study depth completion using deep image prior (DIP) technique which we call deep depth prior. Given color images and noisy and incomplete depth maps, we optimize a randomly-initialized CNN model to reconstruct the depth map using the CNN network structure as a prior. We also use a view-constrained photo- consistency loss, computed using images from a geometrically calibrated camera from nearby viewpoints. It is optimized on test data, so it is independent of training data distributions. We apply this deep depth prior for inpainting and refining incomplete and noisy depth maps within both binocular and multi-view stereo pipelines.
Examining Committee:

Chair: Dr. Larry S. Davis Dean's rep: Dr. Behtash Babadi Members: Dr. Abhinav Shrivastava
Dr. David Jacobs Dr. Soheil Feizi