Advanced Techniques in
Visual Learning & Recognition

CMSC828I - Fall 2020 · University of Maryland

Overview

A graduate-level course in computer vision, with an emphasis on high-level recognition tasks. We will read an eclectic mix of classic and contemporary papers on a wide-range of topics. The course structure will combine lectures, in-class discussions, assignments, and a course project. The goal of this course is to:

Become knowledgeable about how a particular sub-topic evolved, the state-of-the-art, and the challenges that need to be addressed.
Develop skills to critically read, analyze, and discuss a body of research.
Learn to draw connections between different works and engage in productive discussions.

Prerequisites

While there are no formal prerequisites for this course, familiarity with introductory courses in computer vision (CMSC426 or similar) and machine learning (CMSC422 or similar) is assumed. If you have not taken courses covering this material, consult with the instructor. Note that a basic knowledge of linear algebra, probability, and calculus is required.

Logistics

Where & when

Online
Tuesday, Thursday 12:30pm - 1:45pm

Instructor

Abhinav Shrivastava
4238 IRB
abhinav@cs.umd.edu
Office hours: TBD.

Teaching Assistant

Pulkit Kumar
pulkit@umd.edu
Office hours: TBD.

Shishira Maiya
shishira@umd.edu
Office hours: TBD.

Naman Awasthi
nawasthi@umd.edu
Office hours: TBD.

Jun Wang
junwang@umiacs.umd.edu
Office hours: TBD.

Quick Links

Piazza
Web Accessibility
CMSC828I Fall 2019
CMSC828I Fall 2018

Schedule

Coming soon!

Date	Topic	Readings	Slides	Notes
I – Background and Foundations
Sept 1	Introduction to the class		slides
Sept 3	Quiz
Sept 8 Sept 10	Introduction to Data	80 million tiny images: a large dataset for non-parametric object and scene recognition, PAMI 2008 Visual long-term memory has a massive storage capacity for object details, PNAS 2008 Unreasonable Effectiveness of Data, 2009 Unbiased Look at Dataset BiasTorralba & Efros, CVPR 2011	slides
Sept 15 Sept 17	Data-driven methods in vision	Image Restoration using Online Photo Collections , ICCV 2009 Scene Completion Using Millions of Photographs, SIGGRAPH 2007 im2gps: estimating geographic information from a single image, CVPR 2008 Matching and Predicting Street Level Images, ECCV Workshops 2008 Additional Readings: Data-driven Visual Similarity for Cross-domain Image Matching, SIGGRAPH Asia 2011 Non-parametric Scene Parsing , CVPR 2009 Unsupervised Discovery of Mid-Level Discriminative Patches; ECCV 2012 Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships , NIPS 2009	slides
Sept 17 Sept 22 Sept 24	ConvNets and Architectures	Important Architectures: AlexNet VGG ResNet GoogLeNet/Inception DenseNet ConvNet Survey Architectures for Videos: Two-stream I3D S3D-G What goes on inside a CNN? Visualizing and understanding convolutional networks; ECCV 2014 Deep inside convolutional networks: Visualising image classification models and saliency maps; arXiv 2013 Network Dissection: Quantifying Interpretability of Deep Visual Representations; CVPR 2017	slides	bold = required
II – Core Tasks
Sept 24 Sept 29 Oct 1	Two foundational tasks: Object Detection Image Segmentation	Background - Object Detection: Histograms of Oriented Gradients for Human Detection; CVPR 2015 Object Detection with Discriminatively Trained Part-Based Models; IEEE TPAMI 2009 Background - Segmentation: TextonBoost for Image Understanding ...; IJCV 2007 Superpixels and Supervoxels in an Energy Optimization Framework; ECCV 2010 Single-stage object detection: OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks; ICLR 2014 SSD: Single Shot MultiBox Detector; ECCV 2016 You Only Look Once: Unified, Real-Time Object Detection; CVPR 2016	slides
Oct 6 Oct 8 Oct 13 Oct 15	Two foundational tasks: Object Detection Image Segmentation	Semantic Segmentation: Fully Convolutional Networks for Semantic Segmentation; ICCV 2015 Hypercolumns for Object Segmentation and Fine-grained Localization; CVPR 2015 Learning Deconvolution Network for Semantic Segmentation; ICCV 2015 DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs; TPAMI 2017 Object Proposals: Selective Search for Object Recognition; IJCV 2013 What makes for effective detection proposals?; TPAMI 2015 Multi-stage object detection: Region-based Convolutional Networks for Accurate Object Detection and Semantic Segmentation; TPAMI 2015 Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition; TPAMI 2015 Fast R-CNN; ICCV 2015 Faster R-CNN; NIPS 2015 Multi-stage detection & instance segmentation: Simultaneous Detection and Segmentation; ECCV 2014 R-FCN: Object Detection via Region-based Fully Convolutional Networks; NIPS 2016 Instance-aware Semantic Segmentation via Multi-task Network Cascades; CVPR 2016 Mask R-CNN; ICCV 2017 An awesome overview of recent detection methods -- MMDetection; arXiv 2019	slides
Oct 20 Oct 22	Two foundational tasks: Object Detection Image Segmentation	Architectures: Aggregated Residual Transformations for Deep Neural Networks; CVPR 2017 Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks; CVPR 2016 Beyond Skip Connections: Top-Down Modulation for Object Detection; arxiv 2017 Feature Pyramid Networks for Object Detection; CVPR 2017 Scale-Aware Trident Networks for Object Detection; ICCV 2019 Analysis and diagnosis: Deformable Part Models are Convolutional Neural Networks; CVPR 2015 Speed/accuracy trade-offs for modern convolutional object detectors; CVPR 2017 Diagnosing Error in Object Detectors; ECCV 2012 TIDE: A General Toolbox for Identifying Object Detection Errors ; ECCV 2020 Training: [OHEM] Training region-based object detectors with online hard example mining; CVPR 2016 Focal Loss for Dense Object Detection; ICCV 2017	slides
Oct 22 Oct 27	Introduction to other tasks; Human Pose Estimation	Background Reading: Articulated pose estimation with flexible mixtures-of-parts; CVPR 2011 Readings: Pose Machines: Articulated Pose Estimation via Inference Machines; ECCV 2014 Convolutional Pose Machines; CVPR 2016 Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields; CVPR 2017 Towards Accurate Multi-person Pose Estimation in the Wild; CVPR 2017 Stacked hourglass models for pose detection; ECCV 2016	slides
III – Advanced Tasks
Oct 27 Nov 5	Action Recognition	Background: Action Recognition by Dense Trajectories; CVPR 2011 Primary Readings: ActionVLAD: Learning spatio-temporal aggregation for action classification; CVPR 2017 Gaussian Temporal Awareness Networks for Action Localization; CVPR 2019 Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors; CVPR 2015 PA3D: Pose-Action 3D Machine for Video Recognition; CVPR 2019 FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding; CVPR 2020 Additional Readings: Dynamic Image Networks for Action Recognition; CVPR 2016 PoTion: Pose MoTion Representation for Action Recognition; CVPR 2018 Compressed Video Action Recognition; CVPR 2018 Additional Readings (tasks and architectures): Large-scale Video Classification with Convolutional Neural Networks; CVPR 2014 Two-Stream Convolutional Networks for Action Recognition in Videos; NIPS 2014 Learning Spatiotemporal Features with 3D Convolutional Networks; ICCV 2015 Rethinking Spatiotemporal Feature Learning For Video Understanding; ECCV 2018 R-C3D: Region Convolutional 3D Network for Temporal Activity Detection; ICCV 2017 SST: Single-Stream Temporal Action Proposals; CVPR 2017 Action Tubelet Detector for Spatio-Temporal Action Localization; ICCV 2017 Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos; ICCV 2017	slides
Nov 10	Assortment (Part I): Context Reasoning	Primary Readings: Efficient Object Category Recognition Using Classeme; ECCV 2010 Contextual Priming and Feedback for Faster R-CNN; ECCV 2016 Iterative Visual Reasoning Beyond Convolutions; CVPR 2018 Spatial-aware Graph Relation Network for Large-scale Object Detection; CVPR 2018 Reasoning-RCNN: Unifying Adaptive Global Reasoning into Large-scale Object Detection; ; CVPR 2018 Background and Additional Readings: Auto-context and Its Application to High-level Vision Tasks and 3D Brain Image Segmentation; TPAMI 2010 An Empirical Study of Context in Object Detection; CVPR 2009 Objects in Context; ICCV 2007 Putting Objects in Perspective; CVPR 2006	slides
Nov 12 Nov 17	All about 3D: 3D Scene Understanding - Primitives and Reasoning Objects + 3D	Background: Recovering Intrinsic Scene Characteristics From Images; Computer Vision Systems, 1978 Data-Driven 3D Primitives for Single Image Understanding; ICCV 2013 Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture; ICCV 2015 3D2PM – 3D Deformable Part Models; ECCV 2012 ShapeNet: An Information-Rich 3D Model Repository; arXiv 2015 Required readings: Scene Primitives: Learning Ordinal Relationships for Mid-Level Vision; ICCV 2015 Designing Deep Networks for Surface Normal Estimation; CVPR 2015 Scene Reasoning: Blocks World Revisited; ECCV 2010 Holistic Scene Understanding for 3D Object Detection with RGBD cameras; ICCV 2013 3D Objects: 3D ShapeNets: A Deep Representation for Volumetric Shapes; CVPR 2015 Learning a Predictable and Generative Vector Representation for Objects; ECCV 2016 Learning Shape Abstractions by Assembling Volumetric Primitives; CVPR 2017 Category-Specific Object Reconstruction from a Single Image; CVPR 2015 Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images “In the Wild”; ICCV 2019 3D Humans++: Learning 3D Human Dynamics from Video; CVPR 2019 Predicting 3D Human Dynamics from Video; ICCV 2019 Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild; ECCV 2020 Additional and background readings: Unfolding an Indoor Origami World; ECCV 2014 Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene; CVPR 2018 Learning Category-Specific Mesh Reconstruction from Image Collections; ECCV 2018 ObjectNet3D: A Large Scale Database for 3D Object Recognition; ECCV 2016 Learning NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis; ECCV 2020 Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning; ICCV 2019 Great resource for 3D-related papers: 3D Machine Learning	slides
Nov 20 Nov 24	Image Generative Models	GANs (primary readings): Generative Adversarial Nets Tutorial on GANs (slides) VAEs (primary readings): Tutorial on VAEs Auto-Encoding Variational Bayes Stochastic Backpropagation and Approximate Inference in Deep Generative Models Assortment of Image Generative Models (recommended readings): beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework; ICLR 2017 InfoVAE: Information Maximizing Variational Autoencoders; AAAI 2019 Unsupervised representation learning with deep convolutional generative adversarial networks; ICLR 2016 Deep Convolutional Inverse Graphics Network; NIPS 2015 Image-to-Image Translation with Conditional Adversarial Nets; CVPR 2017 Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks; ICCV 2017 IMUNIT: Multimodal UNsupervised Image-to-image Translation; ECCV 2018 DRIT++: Diverse Image-to-Image Translation via Disentangled Representations; 2018 Progressive Growing of GANs for Improved Quality, Stability, and Variation; ICLR 2018 Assortment of Image Generative Models (additional readings): Semi-parametric Image Synthesis; CVPR 2018 PixelNN: Example-based Image Synthesis; ICLR 2018 Pixel Recurrent Neural Networks; ICML 2016 Learning to Generate Chairs, Tables and Cars with Convolutional Networks; PAMI 2017 On the "steerability" of generative adversarial networks; arXiv 2019 Video-to-Video Synthesis; NeurIPS 2018 Go Crazy	slides (in progress)

Resources

For a comprehensive review of Computer Vision, please refer to "Computer Vision: Algorithms and Applications by Richard Szeliski. The book is available for free online or available for purchase.

We will update this space with computer vision you can refer to.

Tutorials (libraries and computation resources)

Accommodations and Policies

Academic Integrity

Note that academic dishonesty includes not only cheating, fabrication, and plagiarism, but also includes helping other students commit acts of academic dishonesty by allowing them to obtain copies of your work. In short, all submitted work must be your own. Cases of academic dishonesty will be pursued to the fullest extent possible as stipulated by the Office of Student Conduct. It is very important for you to be aware of the consequences of cheating, fabrication, facilitation, and plagiarism. For more information on the Code of Academic Integrity or the Student Honor Council, please visit http://www.shc.umd.edu.

Excused Absence and Academic Accommodations

Any student who needs to be excused for an absence from a single lecture, recitation, or lab due to a medically necessitated absence shall:

Make a reasonable attempt to inform the instructor of his/her illness prior to the class.
Upon returning to the class, present their instructor with a self-signed note attesting to the date of their illness. Each note must contain an acknowledgment by the student that the information provided is true and correct. Providing false information to University officials is prohibited under Part 9(i) of the Code of Student Conduct (V-1.00(B) University of Maryland Code of Student Conduct) and may result in disciplinary action.
This self-documentation may not be used for the Major Scheduled Grading Events as defined below.

Any student who needs to be excused for a Major Scheduled Grading Event, must provide written documentation of the illness from the Health Center or from an outside health care provider. This documentation must verify dates of treatment and indicate the time frame that the student was unable to meet academic responsibilities. No diagnostic information shall be given. The Major Scheduled Grading Events for this course include midterm and final exam. For class presentations, the instructor will help the student swap their presentation slot with other students.

It is also the student's responsibility to inform the instructor of any intended absences from exams and class presentations for religious observances in advance. Notice should be provided as soon as possible, but no later than the Monday prior to the the midterm exam, the class presentation date, and the final exam.

Any student eligible for and requesting reasonable academic accommodations due to a disability is requested to provide a letter of accommodation from the Office of Disability Support Services within the first three weeks of the semester.

Other Accommodations and Policies

You can find the university’s course policies here.

Advanced Techniques in Visual Learning & Recognition