Advanced Techniques in
Visual Learning & Recognition

CMSC828I - Fall 2021 · University of Maryland

Overview

A graduate-level course in computer vision, with an emphasis on high-level recognition tasks. We will read an eclectic mix of classic and contemporary papers on a wide-range of topics. The course structure will combine lectures, in-class discussions, assignments, and a course project. The goal of this course is to:

Become knowledgeable about how a particular sub-topic evolved, the state-of-the-art, and the challenges that need to be addressed.
Develop skills to critically read, analyze, and discuss a body of research.
Learn to draw connections between different works and engage in productive discussions.

Student Masking

According to the University’s COVID-19 compliance guidelines, any student with an approved COVID-19 vaccination exemption must wear a mask at all times while indoors and outdoors when around others. These students must also be tested twice each week for COVID-19 and must sign the University Health Center’s Memorandum to Unvaccinated Individuals. Additionally, current County and University guidelines require all individuals to wear a mask indoors. Any student not abiding by these expectations may be in violation of the Code of Student Conduct, Part 10(e)(3): Failure to comply with a directive of University officials.

Prerequisites

While there are no formal prerequisites for this course, familiarity with introductory courses in computer vision (CMSC426 or similar) and machine/deep learning learning (CMSC422 or similar) is assumed. If you have not taken courses covering this material, consult with the instructor. Note that a basic knowledge of linear algebra, probability, and calculus is required.

Logistics

Where & when

CHE 2110
Tuesday, Thursday 3:30pm - 4:45pm

Final Exam

Tuesday, December 21 10:30am-12:30pm

Instructor

Abhinav Shrivastava
4238 IRB
abhinav@cs.umd.edu
Office hours: TBD.

Teaching Assistant

Matthew Gwilliam
mgwillia@umd.edu
Office hours: TBD.

Hanyu Wang
hywang66@umd.edu
Office hours: TBD.

Quick Links

Piazza
Web Accessibility
CMSC828I Fall 2020
CMSC828I Fall 2019
CMSC828I Fall 2018

Schedule

Date	Topic	Readings	Slides	Notes
I – Background and Foundations
Aug 31	Introduction to the class		slides
Sep 2	Quiz
Sep 7 Sep 9	Introduction to Data	80 million tiny images: a large dataset for non-parametric object and scene recognition, PAMI 2008 Visual long-term memory has a massive storage capacity for object details, PNAS 2008 Unreasonable Effectiveness of Data, 2009 Unbiased Look at Dataset BiasTorralba & Efros, CVPR 2011	slides	Discuss add/drop with the instructor.
Sep 14	Data-driven methods in vision	Image Restoration using Online Photo Collections , ICCV 2009 Scene Completion Using Millions of Photographs, SIGGRAPH 2007 im2gps: estimating geographic information from a single image, CVPR 2008 Matching and Predicting Street Level Images, ECCV Workshops 2008 Additional Readings: Data-driven Visual Similarity for Cross-domain Image Matching, SIGGRAPH Asia 2011 Non-parametric Scene Parsing , CVPR 2009 Unsupervised Discovery of Mid-Level Discriminative Patches; ECCV 2012 Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships , NIPS 2009	slides
Sep 16 Sep 21	ConvNets and Architectures	Important Architectures: AlexNet VGG ResNet GoogLeNet/Inception DenseNet ConvNet Survey Architectures for Videos: Two-stream I3D S3D-G What goes on inside a CNN? Visualizing and understanding convolutional networks; ECCV 2014 Deep inside convolutional networks: Visualising image classification models and saliency maps; arXiv 2013 Network Dissection: Quantifying Interpretability of Deep Visual Representations; CVPR 2017 Latest & greatest: Transformers ViT: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale; ICLR 2021 DeIT: Focal Self-attention for Local-Global Interactions in Vision Transformers; 2021 NesT: Aggregating Nested Transformers; 2021 MViT: Multiscale Vision Transformers; 2021	slides	bold = required
II – Core Tasks
Sep 23 Sep 28 Sep 30	Two foundational tasks: Object Detection Image Segmentation	Background - Object Detection: Histograms of Oriented Gradients for Human Detection; CVPR 2015 Object Detection with Discriminatively Trained Part-Based Models; IEEE TPAMI 2009 Background - Segmentation: TextonBoost for Image Understanding ...; IJCV 2007 Superpixels and Supervoxels in an Energy Optimization Framework; ECCV 2010	slides
Sep 30 Oct 5	Two foundational tasks: Object Detection Image Segmentation	Single-stage object detection: OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks; ICLR 2014 SSD: Single Shot MultiBox Detector; ECCV 2016 You Only Look Once: Unified, Real-Time Object Detection; CVPR 2016 Semantic Segmentation: Fully Convolutional Networks for Semantic Segmentation; ICCV 2015 Hypercolumns for Object Segmentation and Fine-grained Localization; CVPR 2015 Learning Deconvolution Network for Semantic Segmentation; ICCV 2015 DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs; TPAMI 2017 Object Proposals: Selective Search for Object Recognition; IJCV 2013 What makes for effective detection proposals?; TPAMI 2015	slides	Assignment 1 out (Oct 5)
Oct 5 Oct 7 Oct 12	Two foundational tasks: Object Detection Image Segmentation	Multi-stage object detection: Region-based Convolutional Networks for Accurate Object Detection and Semantic Segmentation; TPAMI 2015 Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition; TPAMI 2015 Fast R-CNN; ICCV 2015 Faster R-CNN; NIPS 2015 Multi-stage detection & instance segmentation: Simultaneous Detection and Segmentation; ECCV 2014 R-FCN: Object Detection via Region-based Fully Convolutional Networks; NIPS 2016 Instance-aware Semantic Segmentation via Multi-task Network Cascades; CVPR 2016 Mask R-CNN; ICCV 2017 Transformers for detection/segmentation: End-to-End Object Detection with Transformers; ECCV 2020 Deformable DETR: Deformable Transformers for End-to-End Object Detection ; ICLR 2021 Architectures: Aggregated Residual Transformations for Deep Neural Networks; CVPR 2017 Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks; CVPR 2016 Beyond Skip Connections: Top-Down Modulation for Object Detection; arxiv 2017 Feature Pyramid Networks for Object Detection; CVPR 2017 Scale-Aware Trident Networks for Object Detection; ICCV 2019 Analysis and diagnosis: Deformable Part Models are Convolutional Neural Networks; CVPR 2015 Speed/accuracy trade-offs for modern convolutional object detectors; CVPR 2017 Diagnosing Error in Object Detectors; ECCV 2012 TIDE: A General Toolbox for Identifying Object Detection Errors ; ECCV 2020 Training: [OHEM] Training region-based object detectors with online hard example mining; CVPR 2016 Focal Loss for Dense Object Detection; ICCV 2017 An awesome overview of recent detection methods -- MMDetection; arXiv 2019	slides	Assignment 1 due (Oct 12)
III – Additional Topics
Oct 12 Oct 14	Introduction to other tasks; Human Pose Estimation	Background Reading: Articulated pose estimation with flexible mixtures-of-parts; CVPR 2011 Readings: Pose Machines: Articulated Pose Estimation via Inference Machines; ECCV 2014 Convolutional Pose Machines; CVPR 2016 Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields; CVPR 2017 Towards Accurate Multi-person Pose Estimation in the Wild; CVPR 2017 Stacked hourglass models for pose detection; ECCV 2016	slides
Oct 19	Introduction to Reinforcement Learning (Mara Levy)
Oct 21 - Mid-term Exam
IV – Guest Lectures
Oct 26	Self-supervised Learning (Ishan Misra)	Context from images: Unsupervised Visual Representation Learning by Context Prediction; ICCV 2015 Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles; ECCV 2016 Self-Supervised Learning of Pretext-Invariant Representations; CVPR 2020 A Simple Framework for Contrastive Learning of Visual Representations; ICML 2020 Emerging Properties in Self-Supervised Vision Transformers; ICCV 2021 Barlow Twins: Self-Supervised Learning via Redundancy Reduction; ICML 2021 BEiT: BERT Pre-Training of Image Transformers; 2021 Context from videos: Learning Features by Watching Objects Move; CVPR 2017 Transitive Invariance for Self-supervised Visual Representation Learning; ICCV 2017 Learning Correspondence from the Cycle-Consistency of Time; CVPR 2019
Oct 28	Learned Compression: Beyond Images and Video (Saurabh Singh)	Background: Nonlinear Transform Coding; IEEE 2020 End-to-end optimized image compression; ICLR 2017 Variational image compression with a scale hyperprior; ICLR 2018 Applications to domains: End-to-end learning of compressible features; ICIP 2020 Scalable model compression by entropy penalized reparameterization; ICLR 2020 3D Scene compression through entropy penalized neural representation functions; pre-print 2021 Deep implicit volume compression; CVPR 2020
Nov 2	Neural Architecture Search (Debadeepta Dey)	Background: Neural Architecture Search: A Survey; JMLR 2019 One-shot NAS: ENAS: Efficient Neural Architecture Search via Parameter Sharing; ICML 2018 DARTS: Differentiable Architecture Search; ICLR 2019 ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware; ICLR 2019 Once-for-All: Train One Network and Specialize it for Efficient Deployment; ICLR 2020 Discrete Search: Neural Architecture Search with Reinforcement Learning; ICLR 2017 Exploring the Loss Landscape in Neural Architecture Search; UAI 2021 BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search; AAAI 2021 Hybrid of one-shot and discrete: Petridish: Efficient Forward Architecture Search; NeurIPS 2019 Program synthesis style search: Primer: Searching for Efficient Transformers for Language Modeling; pre-print 2021
Nov 4	Vision Techniques for Reinforcement Learning (Mary Levy)	Background: On Evaluation of Embodied Navigation Agents; arxiv 2018 Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey; IEEE Symposium 2020 Sim-to-Real Transfer: Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency; ICLR 2021 Oral RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real; CVPR 2020 Domain Randomization and Generative Models for Robotic Grasping; IROS 2018 Visual Navigation: Semantic Visual Navigation by Watching YouTube Videos; NeurIPS 2020 Neural Topological SLAM for Visual Navigation; CVPR 2020 Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning; ICRA 2017 Learning Navigation Subroutines from Egocentric Videos; PMLR 2020
Nov 9	Action Recognition	Background: Action Recognition by Dense Trajectories; CVPR 2011 Primary Readings: ActionVLAD: Learning spatio-temporal aggregation for action classification; CVPR 2017 Gaussian Temporal Awareness Networks for Action Localization; CVPR 2019 Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors; CVPR 2015 PA3D: Pose-Action 3D Machine for Video Recognition; CVPR 2019 FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding; CVPR 2020 Additional Readings: Dynamic Image Networks for Action Recognition; CVPR 2016 PoTion: Pose MoTion Representation for Action Recognition; CVPR 2018 Compressed Video Action Recognition; CVPR 2018 Additional Readings (tasks and architectures): Large-scale Video Classification with Convolutional Neural Networks; CVPR 2014 Two-Stream Convolutional Networks for Action Recognition in Videos; NIPS 2014 Learning Spatiotemporal Features with 3D Convolutional Networks; ICCV 2015 Rethinking Spatiotemporal Feature Learning For Video Understanding; ECCV 2018 R-C3D: Region Convolutional 3D Network for Temporal Activity Detection; ICCV 2017 SST: Single-Stream Temporal Action Proposals; CVPR 2017 Action Tubelet Detector for Spatio-Temporal Action Localization; ICCV 2017 Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos; ICCV 2017	slides
Nov 18	Attributes	Background: Visual Attributes in the Wild (VAW); CVPR 2021 Describing Objects by their Attributes; CVPR 2009 Large-Scale Object Classification Using Label Relation Graphs; ECCV 2014 Discovering States and Transformations in Image Collections; CVPR 2015 From Red Wine to Red Tomato: Composition with Context; CVPR 2017	slides
Nov 23	Context Reasoning	Primary Readings: Efficient Object Category Recognition Using Classeme; ECCV 2010 Contextual Priming and Feedback for Faster R-CNN; ECCV 2016 Iterative Visual Reasoning Beyond Convolutions; CVPR 2018 Spatial-aware Graph Relation Network for Large-scale Object Detection; CVPR 2018 Reasoning-RCNN: Unifying Adaptive Global Reasoning into Large-scale Object Detection; ; CVPR 2018 Background and Additional Readings: Auto-context and Its Application to High-level Vision Tasks and 3D Brain Image Segmentation; TPAMI 2010 An Empirical Study of Context in Object Detection; CVPR 2009 Objects in Context; ICCV 2007 Putting Objects in Perspective; CVPR 2006	slides
Nov 23 Nov 30 Dec 2	All about 3D: 3D Scene Understanding - Primitives and Reasoning Objects + 3D	Background: Recovering Intrinsic Scene Characteristics From Images; Computer Vision Systems, 1978 Data-Driven 3D Primitives for Single Image Understanding; ICCV 2013 Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture; ICCV 2015 3D2PM – 3D Deformable Part Models; ECCV 2012 ShapeNet: An Information-Rich 3D Model Repository; arXiv 2015 Required readings: Scene Primitives: Learning Ordinal Relationships for Mid-Level Vision; ICCV 2015 Designing Deep Networks for Surface Normal Estimation; CVPR 2015 Scene Reasoning: Blocks World Revisited; ECCV 2010 Holistic Scene Understanding for 3D Object Detection with RGBD cameras; ICCV 2013 3D Objects 3D ShapeNets: A Deep Representation for Volumetric Shapes; CVPR 2015 Learning a Predictable and Generative Vector Representation for Objects; ECCV 2016 Learning Shape Abstractions by Assembling Volumetric Primitives; CVPR 2017 Category-Specific Object Reconstruction from a Single Image; CVPR 2015 Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images “In the Wild”; ICCV 2019 An End-to-End Transformer Model for 3D Object Detection; ICCV 2021 Additional and background readings: Unfolding an Indoor Origami World; ECCV 2014 Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene; CVPR 2018 Learning Category-Specific Mesh Reconstruction from Image Collections; ECCV 2018 ObjectNet3D: A Large Scale Database for 3D Object Recognition; ECCV 2016 Great resource for 3D-related papers: 3D Machine Learning	slides
Dec 2 Dec 7 Dec 9	Generative Models	Primary Readings: GANs: Generative Adversarial Nets Tutorial on GANs (slides) VAEs: Tutorial on VAEs Auto-Encoding Variational Bayes Stochastic Backpropagation and Approximate Inference in Deep Generative Models Additional Readings (Assortment of Image Generative Models): beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework; ICLR 2017 InfoVAE: Information Maximizing Variational Autoencoders; AAAI 2019 Unsupervised representation learning with deep convolutional generative adversarial networks; ICLR 2016 Deep Convolutional Inverse Graphics Network; NIPS 2015 Image-to-Image Translation with Conditional Adversarial Nets; CVPR 2017 Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks; ICCV 2017 Semi-parametric Image Synthesis; CVPR 2018 PixelNN: Example-based Image Synthesis; ICLR 2018 Pixel Recurrent Neural Networks; ICML 2016 Learning to Generate Chairs, Tables and Cars with Convolutional Networks; PAMI 2017 On the "steerability" of generative adversarial networks; arXiv 2019 Video-to-Video Synthesis; NeurIPS 2018 Go Crazy	slides
Dec 9	Misc. + Epilogue + Ethics	Suggested Self-readings for Visual data mining and discovery: Unsupervised Discovery of Mid-Level Discriminative Patches; ECCV 2012 Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States; PNAS 2017 GeoStyle: Discovering Fashion Trends and Events; ICCV 2019 Additional readings: What Makes Paris Look like Paris?; SIGGRAPH 2012 Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time; ICCV 2013 Where have all the people gone? Enhancing global conservation using night lights and social media; Ecological Applications 2015 Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images; ICCV 2017 Visualizing Brand Associations from Web Community Photos; ACM WSDM 2014 Readings on other topics, please refer to Fall 2019 website.	slides
Final Exam: Dec 21 10:30am - 12:30pm

Resources

For a comprehensive review of Computer Vision, please refer to "Computer Vision: Algorithms and Applications by Richard Szeliski. The book is available for free online or available for purchase.

We will update this space with computer vision you can refer to.

Tutorials (libraries and computation resources)

Accommodations and Policies

Academic Integrity

Note that academic dishonesty includes not only cheating, fabrication, and plagiarism, but also includes helping other students commit acts of academic dishonesty by allowing them to obtain copies of your work. In short, all submitted work must be your own. Cases of academic dishonesty will be pursued to the fullest extent possible as stipulated by the Office of Student Conduct. It is very important for you to be aware of the consequences of cheating, fabrication, facilitation, and plagiarism. For more information on the Code of Academic Integrity or the Student Honor Council, please visit http://www.shc.umd.edu.

Excused Absence and Academic Accommodations

Any student who needs to be excused for an absence from a single lecture, recitation, or lab due to a medically necessitated absence shall:

Make a reasonable attempt to inform the instructor of his/her illness prior to the class.
Upon returning to the class, present their instructor with a self-signed note attesting to the date of their illness. Each note must contain an acknowledgment by the student that the information provided is true and correct. Providing false information to University officials is prohibited under Part 9(i) of the Code of Student Conduct (V-1.00(B) University of Maryland Code of Student Conduct) and may result in disciplinary action.
This self-documentation may not be used for the Major Scheduled Grading Events as defined below.

Any student who needs to be excused for a Major Scheduled Grading Event, must provide written documentation of the illness from the Health Center or from an outside health care provider. This documentation must verify dates of treatment and indicate the time frame that the student was unable to meet academic responsibilities. No diagnostic information shall be given. The Major Scheduled Grading Events for this course include midterm and final exam. For class presentations, the instructor will help the student swap their presentation slot with other students.

It is also the student's responsibility to inform the instructor of any intended absences from exams and class presentations for religious observances in advance. Notice should be provided as soon as possible, but no later than the Monday prior to the the midterm exam, the class presentation date, and the final exam.

Any student eligible for and requesting reasonable academic accommodations due to a disability is requested to provide a letter of accommodation from the Office of Disability Support Services within the first three weeks of the semester.

Other Accommodations and Policies

You can find the university’s course policies here.

Advanced Techniques in Visual Learning & Recognition