CMSC 828W: Foundations of Deep Learning

Welcome to the CMSC 828W course on Foundations of Deep Learning for Fall 2021.

CMSC 828W is currently being taught by Soheil Feizi.

Announcements

• Lecture 1 (8/31): DL basics: ERM and basic architectures (MLP, ResNet and CNNs), overview of SGD, ADAMs, Momentum, etc.
  Readings:

  • The Deep Learning Book

• Lecture 2 (9/2): DL optimization: SGD convergence in overparameterized regime.
  Readings:

  • Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

• Lecture 3 (9/7): Cont. DL optimization: SGD convergence in overparameterized regime.
  Readings:

  • Loss landscapes and optimization in over-parameterized non-linear systems and neural networks

  • Additional Reading: Gradient Descent Provably Optimizes Over-parameterized Neural Networks

• Lecture 4 (9/9): DL optimization, Implicit Bias of GD
  

  • The Implicit Bias of Gradient Descent on Separable Data

• Lecture 5 (9/14): Cont. DL optimization, Implicit Bias of GD
  

  • The Implicit Bias of Gradient Descent on Separable Data

• Lecture 6 (9/16): DL generalization: Re-thinking generalization measures + double descent.
  Readings:

  • Understanding Machine Learning: From Theory to Algorithms

  • Understanding deep learning requires rethinking generalization

  • Reconciling modern machine learning practice and the bias-variance trade-off

• Lecture 7 (9/21): Neural tangent kernels
  Readings:

  • Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  • A Note on Lazy Training in Supervised Differentiable Programming

• Lecture 8 (9/28): DL robustness: Adversarial attacks and defenses.
  Readings:

  • Towards Deep Learning Models Resistant to Adversarial Attacks

  • Towards Evaluating the Robustness of Neural Networks

  • Do ImageNet Classifiers Generalize to ImageNet?

• Lecture 9 (9/30): DL robustness: Gradient Masking + Are adversarial examples inevitable?
  Readings:

  • Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  • Are adversarial examples inevitable?

• Lecture 10 (10/5): DL robustness: Provable and Generalizable Defenses
  Slides: click here
  Readings:

  • Certified Adversarial Robustness via Randomized Smoothing

  • Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness

  • Improved, Deterministic Smoothing for L1 Certified Robustness

  • Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablation

  • Perceptual Adversarial Robustness: Defense Against Unseen Threat Models

• Lecture 11 (10/7): DL Robustness: Adversarial Poisoning Attacks and Defenses
  Readings:

  • Clean-Label Backdoor Attacks

  • Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

  • Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks

  • Deep Partition Aggregation: Provable Defense against General Poisoning Attacks

• Lecture 12 (10/8): Deep generative models: VAEs.
  Readings:

  • Auto-Encoding Variational Bayes

  • β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  • Neural Discrete Representation Learning

  • Generating Diverse High-Fidelity Images with VQ-VAE-2

  • NVAE: A Deep Hierarchical Variational Autoencoder

• Lecture 13 (10/13): Deep generative models: GANs.
  Readings:

  • Generative Adversarial Nets

  • Wasserstein GAN

  • Spectral Normalization for Generative Adversarial Networks

  • Large Scale GAN Training for High Fidelity Natural Image Synthesis

  • GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

  • Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation

  • Understanding GANs in the LQG setting

• Lecture 14 (10/19): Deep generative models: GANs (part II).

• Lecture 15 (10/21): Midpoint Project Presentations (Part I)

• Lecture 16 (10/26): Midpoint Project Presentations (Part II)

• Lecture 17 (10/28): Deep generative models: Flow-based models.
  Readings:

  • Density estimation using Real NVP

  • Glow: Generative Flow with Invertible 1×1 Convolutions

  • Do Deep Generative Models Know What They Don't Know?

  • Why Normalizing Flows Fail to Detect Out-of-Distribution Data?

  • Adversarial Robustness of Flow-Based Generative Models

• Lecture 18 (11/2): Relaxing i.i.d.: Domain adaptation.
  Readings:

  • Analysis of Representations for Domain Adaptation

  • Unsupervised Domain Adaptation by Backpropagation

  • CYCADA: Cyle-Consistent Adversarial Domain Adaptation

  • Impossibility Theorems for Domain Adaptation

• Lecture 19 (11/4): Relaxing i.i.d.: Domain generalization.
  Readings:

  • In Search of Lost Domain Generalization

  • The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

  • The Risks of Invariant Risk Minimization

  • Learning to Generalize: Meta-Learning for Domain Generalization

• Lecture 20 (11/9): Contrastive, self-supervised DL
  Readings:

  • A Simple Framework for Contrastive Learning of Visual Representations

  • A Theoretical Analysis of Contrastive Unsupervised Representation Learning

  • Contrastive learning, multi-view redundancy, and linear models

• Lecture 21 (11/11): Meta-learning
  Readings:

  • Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

  • Meta-Learning with Differentiable Convex Optimization

  • Prototypical Networks for Few-shot Learning

  • On First-Order Meta-Learning Algorithms

• Lecture 22 (11/16): LSTMs and transformers
  Readings:

  • Long Short-Term Memory

  • Bidirectional recurrent neural networks

  • Attention Is All You Need

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Lecture 23 (11/18): Explainable DL: Interpretation methods.
  Readings:

  • How to Explain Individual Classification Decisions

  • A Unified Approach to Interpreting Model Predictions

  • Axiomatic Attribution for Deep Networks

  • SmoothGrad: removing noise by adding noise

  • Sanity Checks for Saliency Maps

  • Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation

• Lecture 24 (11/23): Deep RL
  Readings:

  • Proximal Policy Optimization Algorithms

  • Deep Reinforcement Learning with Double Q-learning

  • Policy Gradient Methods for Reinforcement Learning with Function Approximation

  • Human-level control through deep reinforcement learning

  • Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

• Lecture 25 (11/25): Thanksgiving

• Lecture 26 (11/30): Cont. Deep RL

• Lecture 27 (12/2): Project Presentations (Part I)

• Lecture 28 (12/7): Project Presentations (Part II)

• Lecture 29 (12/9): Final Exam