CMSC 818V · Machine Learning for Physical Sensing & Perception

Overview

This graduate course explores machine learning techniques for physical sensing and perception in real‑world environments: beamforming and DoA; radar/SAR and spectral analysis; localization and sensor fusion (with IMUs); volumetric rendering (NeRF, Gaussian Splatting); room/underwater acoustics; and LLM‑grounded multimodal perception. Learning is through lectures, paper discussions, hands‑on assignments, and a team project.

Prerequisite: Graduate standing (MS/PhD) or permission of instructor.
Modality: In‑person; Tu–Th 3:30–4:45pm, Iribe 1207.
Readings: Research papers & tutorials (see Detailed Syllabus).

Grading Breakdown

Assessment components and weights.
Component	Weight	Notes
Term Project (x1)	50%	One semester-long project completed in groups of two students. Evaluation will be based on the project report (30%) and the end-semester project presentation (20%).
Paper presentation (x1)	20%	One in-class presentation of a scientific article.
Programming Assignment (x1)	15%	One programming assignment in a group of two students.
MCQ Quiz (x2)	10%	Two in-class multiple choice questions quizzes.
Attendance	5%	Attendance and participation in class discussions.

Topics by Week (High‑Level)

Wk 1: Signals & spectra · Beamforming/SAR basics
Wk 2: ML/DL foundations · CNN/RNN tooling
Wk 3: DoA & array processing · Neural beamforming
Wk 4: ToF/Range/Doppler · Deep radar/gestures · Localization (classical → deep)
Wk 5: Source separation (ICA/NMF → Conv‑TasNet/SepFormer)
Wk 6: Tomographic imaging (classical → unrolled nets)
Wk 7: SAR imaging (classical → deep)
Wk 8: Spectrum analysis & wavelets → learnable spectral front‑ends
Wk 9: Room acoustics → SIREN/NeRF/Gaussian Splatting
Wk 10: Transformers & LLMs for sensing
Wk 11: Fusion: Kalman/Particle → multimodal deep fusion
Wk 12: CCA & statistical fusion → self‑supervised multimodal
Wk 13: IMU & motion sensing → deep IMU analytics
Wk 14: Underwater sensing/navigation → deep underwater
Wk 15: Final presentations & demos

See the full, detailed syllabus with readings and applications.

Schedule: Presentations, Quizzes, Project, Submissions

Topics	Date	Materials
0 Course overview	Sep 2	Class Slides
1_1 Signals and sepectral analysis basics	Sep 4	Class Slides Working principles of sensors (refer slides) Signals - definition and types (refer slides) Complex model for signals (Quadrature signals) [Optional] (Multivariable Vector-Valued Functions)
1_2 Frequency Domain View	Sep 9	Class Slides Spectrogram - Time-Frequency view of signals (refer slides) The idea of sampling (refer slides) Signals as vectors and change in bases in vector spaces. (refer slides) [Reference] (3Blue1Brown Tutorial on Change of basis)
1_3 Frequency Domain View	Sep 16	Class Slides
1_4 Frequency in learning	Sep 23	Class Slides
1_5 Frequency in learning	Sep 25	Class Slides
1_6 Frequency in learning	Sep 30	Class Slides
2_1 Inverse Problems in Sensing/Perception	Oct 2	Class Slides
Submit project proposal. Due Oct 16th, 11:59 pm
Programming homework released. Due Oct 20th, 11:59 pm
2_2 Inverse Problems: Spatial Perception	Oct 7	Class Slides
2_3 Inverse Problems: Spatial Perception	Oct 9	Class Slides
Fall break	Oct 14
2_4 Inverse Problems: Spatial Perception: DoA	Oct 16	Class Slides
2_5 Inverse Problems: Spatial Perception: DoA Subspace algorithm	Oct 21	Class Slides
TBD	Oct 23	Class Slides
TBD	Oct 28	Class Slides
TBD	Oct 30	In-class Quiz (Open books/notes/computer, but closed Internet/AI)
	Nov 4	Student presentation: Mahima Beltur & Sungjin Hwang Paper: VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem
	Nov 6	Student presentation: Danyil Butkovskyi & Daniel Pitzele Paper: SoundSpaces: Audio-Visual Navigation in 3D Environments
	Nov 11	Student presentation: Tingfang Cheng & Eileen Yuan Paper: TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models
	Nov 13	Student presentation: Carwyn Collinsworth & Kalonji Harrington  Paper: Adversarial attacks on deep-learning-based radar range profile target recognition
	Nov 18	Student presentation: Aritrik Ghosh & Daniel Lian Paper: DeepSense: Fast Wideband Spectrum Sensing Through Real-Time In-the-Loop Deep Learning
	Nov 20	Student presentation: Max Li  Paper: Deep Non-Line-Of-Sight Reconstruction
	Nov 25	Student presentation: Alec Luterman & Alexander Teacu Paper: PointPainting: Sequential Fusion for 3D Object Detection
Thanksgiving break	Nov 27
TBD	Dec 2	Class Slides
	Dec 4	Student presentation-1: Aditya Shelke & Bruce Gao Paper 1: Through-Wall Human Pose Estimation Using Radio Signals Student presentation-2: Garner Thompson Paper 2: RadarSplat: Radar Gaussian Splatting for High-Fidelity Data Synthesis and 3D Reconstruction of Autonomous Driving Scenes
	Dec 9	In-class Quiz (Open books/notes/computer, but closed Internet/AI) Student presentation-1: Jincheng Yang & Phuc Nguyen Paper 1: DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras Student presentation-2: Yisheng Zhang & Yangsheng Xu Paper 2: Deep Learning for Size-Agnostic Inverse Design of Random-Network 3D Printed Mechanical Metamaterials
Final Project Presentations	Dec 11

Detailed Syllabus

This section mirrors the full syllabus, including topics, study materials, and example applications.

Week 1

Signals and Spectral Basics

Topics: Signal types; Fourier Transform; Spectrum; Nyquist.
Applications: Speech/audio analysis; Wireless spectrum analysis.

Classical Estimation & Imaging

Topics: Beamforming; inverse problems; SAR basics.
Applications: Smart speakers; Earth observation SAR.

Week 2

Introduction to Machine Learning

Topics: Supervised/unsupervised; regression; backpropagation.
Applications: Classification; Regression in sensing.

Deep Learning Foundations

Topics: CNNs; RNNs/LSTMs; PyTorch/Keras.
Applications: Image/audio classification; Sequential IMU modeling.

Week 3

DoA and Array Processing

Topics: Array geometry; MUSIC; ESPRIT.
Applications: Conferencing localization.

Neural DoA & Beamforming

Topics: Deep DoA models; Neural beamforming.
Applications: AR glasses; Drone arrays.

Week 4

ToF, Range, Doppler

Topics: Cross‑correlation; matched filtering; Doppler.
Applications: Automotive radar; Speed detection.

Deep Range & Doppler

Topics: Neural ToF estimation; radar gesture recognition.
Applications: Gesture control; Human activity.

Localization (Classical)

Topics: Triangulation; fingerprinting; dead reckoning.
Applications: Indoor positioning; Asset tracking.

Deep Learning for Localization

Topics: Neural CSI/RSSI; RF+Vision fusion.
Applications: Robotics navigation; Warehouses.

Week 5

Source Separation (Classical)

Topics: ICA; NMF.
Applications: Blind speech separation; Biomedical signals.

Neural Source Separation

Topics: Conv‑TasNet; SepFormer; Demucs.
Applications: Meeting transcription; Hearing aids.

Week 6

Tomographic Imaging (Classical)

Topics: CT/MRI FBP; iterative solvers.
Applications: Medical imaging; Industrial tomography.

Deep Tomography & Unrolled Nets

Topics: LISTA; MoDL.
Applications: MRI acceleration; CT dose reduction.

Week 7

SAR Imaging (Classical)

Topics: SAR backprojection; matched filtering.
Applications: Remote sensing; Surveillance.

Deep SAR Imaging

Topics: GAN priors; super‑resolution.
Applications: High‑res sensing; Drone mapping.

Week 8

Spectrum Analysis & Wavelets

Topics: Spectrograms; wavelets.
Applications: Speech recognition; Seismology.

Learnable Spectral Front‑Ends

Topics: Neural STFT; SpecAugment.
Applications: Robust ASR; Environmental tagging.

Week 9

Room Acoustics (Classical)

Topics: Impulse response; reverberation.
Applications: XR audio; Architectural design.

Neural Room Acoustics

Topics: SIREN; NeRF; Gaussian Splatting.
Applications: Spatial audio; Telepresence.

Week 10

Transformers for Sensing

Topics: Self‑attention; ViTs.
Applications: Vision sensing; Sequential data.

Large Language Models

Topics: Pretraining; prompting; multimodal LLMs.
Applications: Audio‑text fusion; Vision‑text tasks.

Week 11

Fusion with Kalman & Particle Filters

Topics: State estimation; Bayesian fusion.
Applications: Robotics localization; AR/VR.

Deep Multimodal Fusion

Topics: Audio‑visual; Radar‑camera.
Applications: Autonomous driving; Smart homes.

Week 12

CCA & Statistical Fusion

Topics: Canonical correlation; hypothesis testing.
Applications: Speech enhancement; Cross‑modal tasks.

Self‑Supervised Multimodal Learning

Topics: Contrastive; masked modeling.
Applications: AV speech recognition; Cross‑modal retrieval.

Week 13

IMU and Motion Sensing

Topics: Accelerometer & gyroscope; drift issues.
Applications: Smartphone sensing; AR/VR head tracking.

Deep Learning with IMUs

Topics: Activity recognition; gait analysis.
Applications: Health monitoring; Sports analytics.

Week 14

Underwater Sensing & Navigation (Classical)

Topics: Acoustic propagation; sonar; DVL navigation.
Applications: Submarine navigation; AUVs.

Deep Learning for Underwater Sensing

Topics: Neural sonar; underwater localization; RL navigation.
Applications: Ocean monitoring; Marine robotics.