CMSC 818V · Machine Learning for Physical Sensing & Perception

Overview

This graduate course explores how machine learning can be used to infer physical states, properties, and dynamics from real-world sensor measurements. Rather than treating sensor data as arbitrary inputs, we study how measurements arise through physical phenomena, propagation and interaction, and sensor transduction, and how this physical structure can inform learning and inference. Topics include signal and measurement models, inverse problems, spatial and computational sensing, physics-informed learning, state estimation and dynamics, multimodal perception, and robust generalization, with applications spanning acoustics, RF/radar, vision, inertial, and underwater sensing. Learning is through lectures, research-paper discussions, hands-on assignments, and a team research project.

Prerequisite: Graduate standing (MS/PhD) or permission of instructor.
Modality: In‑person; Tu–Th 12:30–1:45pm, Iribe 2107.
Readings: Research papers & tutorials (see Detailed Syllabus).

Grading Breakdown

Assessment components and weights.
Component	Weight	Notes
Term Project (x1)	50%	One semester-long project completed in groups of two students. Evaluation will be based on the project report (30%) and the end-semester project presentation (20%).
Paper presentation (x1)	20%	One in-class presentation of a scientific article.
Programming Assignment (x1)	15%	One programming assignment in a group of two students.
MCQ Quiz (x2)	10%	Two in-class multiple choice questions quizzes.
Attendance	5%	Attendance and participation in class discussions.

Topics

Physical sensing & measurement models · Propagation, interaction, transduction, noise & uncertainty
Signals as physical measurements · Sampling, spectra, phase, convolution, correlation & time-frequency analysis
Inverse problems for physical inference · Identifiability, conditioning, regularization, sparsity & physical priors
Learning for physical inference · Learned priors, algorithm unrolling, physics-informed learning & system identification
Spatial sensing · Time-of-flight, phase, Doppler, sensor arrays, beamforming & direction finding
Computational imaging & sparse sensing · Tomography, synthetic aperture, compressive sensing & learned reconstruction
Representations for physical signals · Waveforms, spectra, spectrograms, CSI, range-Doppler, point clouds & neural representations
Dynamics & state estimation · State-space models, Kalman/particle filtering, sensor fusion & learned dynamics
Multimodal physical perception · Cross-modal representation, alignment, fusion & self-supervised learning
Generalization in physical sensing · Domain shift, simulation, domain randomization, sim-to-real & physical priors
Acoustic sensing & spatial audio · Source localization, separation, room acoustics & learned acoustic fields
RF, radar & wireless perception · Localization, tracking, imaging & environmental sensing
Neural fields & physical scene representations · Implicit representations, NeRF/SIREN & volumetric sensing
Emerging physical perception · Inertial, underwater, multimodal & language-assisted sensing
Research projects · Presentations & demonstrations

Class Schedule

Topics and study materials will be added before each class session; check back for updates. Holiday rows (no class) are highlighted.

Lecture dates, topics, and downloadable slides.
Date	Topic	Study Materials
Sep 1	Course Overview	0_overview.pptx
Sep 3	From the Physical World to Sensor Data	1_PhysicalWorld_Sensor_Data.pptx
Sep 8	Signals as Physical Measurements	2_signals_as_measurements.pptx
Sep 10	Signals as Physical Measurements (Continued) The "Worm"-up programming assignment has been released on ELMS. Collect your pet bug from the TA.
Sep 15	Inverse Problems in Practice	3_Inverse_Problems_In_Practice.pptx
Sep 17	Inverse Problems in Practice (Continued)
Sep 22	Learning in Inverse problems	4_Inverse_to_Learning.pptx
Sep 24	Learning in Inverse problems (Continued)
Sep 29	Learning in Inverse problems (Continued)
Oct 1	Learned Spatial Inference [Project ideas are released. Prepare to propose your project by Oct 7th.]	5_DoA_as_learned_opt.pptx
Oct 6	Learned Spatial Inference (Continued)
Oct 8	Guest Lecture (Aritrik Ghosh)	MUSIC_to_SubspaceNet.pptx
Oct 13	Fall break — no class
Oct 15	Learned Spatial Inference (Continued)
Oct 20	Learned Spatial Inference (Continued)
Oct 22	Learning and Structured Optimization in Miniature Sensing	6_applications_constrained_opt_in_sensing.pptx
Oct 27	No class — Exam preparation
Oct 29	In-class exam	Summary of Lectures 1-5 PDF version of the lecture slides
Nov 3	Learning and Structured Optimization in Miniature Sensing (Continued)
Nov 5	Neural Fields	7_Neural_Fields_and_3D_Reconstruction.pptx
Nov 10	Computational / Compressive Imaging	Student presentation: Donghwa Kang & Heechan Yoon Paper: Events-to-Video: Bringing Modern Computer Vision to Event Cameras (CVPR 2019)
Nov 12	No class — Conference duty
Nov 17	Neural 3D Reconstruction / Active Sensing	Student presentation-1: CJ Nygard & Gavin Crisologo Paper 1: Integrating Multi-Robot Adaptive Sampling and Informative Path Planning for Spatiotemporal Natural Environment Prediction (IEEE) Student presentation-2: Geoff Marcus & Swathi Annambhotla Paper 2: ActiveNeRF: Learning Where to See with Uncertainty Estimation (ECCV 2022)
Nov 19	Sensing Sounds and Vibrations	Student presentation-1: Abu Horaira Hridhon & Tianyang Chen Paper 1: WaveEar: Exploring a mmWave-based Noise-resistant Speech Sensing for Voice-User Interface (ACM MobiSys 2019) Student presentation-2: Kaushal Janga & Shanthosh Mageswari Paper 2: SonicSense: Object Perception from In-Hand Acoustic Vibration (CoRL 2024)
Nov 24	Learned State Estimation & Multimodal Sensor Fusion	Student presentation-1: Ravidu Hevaganinge & Daniel Chen Paper 1: VibeCheck: Using Active Acoustic Tactile Sensing for Contact-Rich Manipulation (IROS 2025) Student presentation-2: Raymond Chen & Rahul Ravi Vk Paper 2: TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (CVPR 2022)
Nov 26	Thanksgiving — no class
Dec 1	Learning Physical Dynamics / World Models & RF / Wireless Perception	Student presentation-1: Declan Kutscher & Shanmuka Sadhu Paper 1: Learning Latent Dynamics for Planning from Pixels (ICML 2019) Student presentation-2: Ojas Jagtap & Virali Kataria Paper 2: Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data (ECCV 2020) Student presentation-3: Andreas Tzitzikas & George Miao Paper 3: Widar3.0: Zero-Effort Cross-Domain Gesture Recognition with Wi-Fi (MobiSys 2019)
Dec 3	Learned State Estimation	Student presentation: Suhani Chaudhary & Gili Gordiyenko Paper: KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics (IEEE TSP 2022)
Dec 8
Dec 10

Course Policy

Academic Integrity

Conduct codes

Please follow the university's Academic Integrity and Student Conduct Codes. Please also carefully read and abide by the Office of Information Technology's policy regarding acceptable use of IT resources.

As per the university's honor pledge policy, on all work submitted for assessment in this course, students are encouraged to write and sign the following pledge:

"I pledge on my honor that I have not given or received any unauthorized assistance on this assessment."

Honesty

Unless explicitly mentioned otherwise, all work submitted for assessment is to be written individually; cooperation or use of unauthorized materials on projects is a violation of the University's Code of Academic Integrity. Any evidence of this, or of unacceptable use of computer resources, use of unauthorized materials, or cooperation on coursework (i.e., any work submitted for assessment, including exams, homeworks, assignments, and projects), or other possible violations of the Honor Code, will be submitted to the Student Honor Council, which could result in an XF for the course, suspension, or expulsion.

Collaboration

For learning the course concepts, students are welcome to study together or to receive help from anyone else. You may discuss with others the assignment/project requirements, what was discussed in class and in the class web forum, and general syntax errors. When it comes to actually writing an exam or programming assignment/project, other than help from the instructional staff, it must solely and entirely be your own work. Working with another student or individual, or using anyone else's work in any way except as noted in this paragraph, is a violation of the code of academic integrity and will be reported to the Honor Council. You may not discuss the design of any part of a project with anyone except the instructor or teaching assistants.

Grading

Grades will be maintained on Gradescope and ELMS. You are responsible for all material discussed in lecture and discussion section and posted on the class web page, including announcements, deadlines, policies, etc. During the semester we may provide ungraded practice homework exercises and solutions. While we will not collect these exercises, completing them is essential preparation for exams. You may work together on these ungraded homeworks, and you may of course come to office hours for additional help.

Regrading

Any request for reconsideration of any grading on coursework must be submitted within one week of when it is returned. Exam regrading requests must be made in writing. Any coursework submitted for reconsideration may be regraded in its entirety, which could result in a lower score if warranted.

Final grade calculation

Your final course grade is determined by the same weights shown in the Grading Breakdown on the Home page: Term Project (50%), Paper presentation (20%), Programming Assignment (15%), MCQ Quiz (10%), and Attendance (5%).

Late submission

Late submissions are strongly discouraged. Unless otherwise mentioned explicitly, coursework submitted after the deadline will incur a 20% deduction on the earned score for every 24-hour period it is late. No submissions will be accepted beyond five days (120 hours) past the deadline.

The adjusted grade for a late submission is calculated as follows:

G_late = G_earned × (1 − 0.2 × ⌈ h / 24 ⌉)

where G_earned and G_late are the grades before and after the penalty, respectively, and h is the number of hours late. Plan accordingly to avoid late penalties.

Attendance, Absence, and Deadline Extension

Class attendance

You are encouraged to attend class and regularly follow the discussed topics. There will be a lot of material taught in class and often written during discussions (outside published slides). So it is in everyone's best interest to attend and engage during lectures.

Exam schedule and coursework deadlines

You are required to attend scheduled exams and submit coursework on time. There are several excused absences from an exam and coursework deadline extension: (a) illness, (b) religious observation, (c) participation in required university activities and academic duties, or (d) a family or personal emergency. We will work with you to make sure that you have a fair amount of time to make up for excused absences and deadline extensions.

Application for excused absences

The best way that we can help is if we know about absences as far in advance as possible.

Provide a request for absence or deadline extension in writing/email.
Provide appropriate documentation that shows the absence qualifies as excused.
Provide as much advance notice as is possible, safe, and appropriate.

Accommodations & Evaluations

Academic accommodations (Accessibility and Disability Service, ADS)

Any student eligible for and requesting reasonable academic accommodations due to a disability is requested to provide, to one of the professors in office hours, a letter of accommodation from the Office of Accessibility and Disability Services (ADS) within the first two weeks of the semester.

All arrangements for exam accommodations as a result of disability must be made and arranged with the instructor at least three business days prior to the exam date; later requests (including retroactive ones) will be refused.

Course evaluations

If you have a suggestion for improving this class, don't hesitate to tell us or the TAs during the semester. At the end of the semester, please don't forget to provide your feedback using the campus-wide CourseEvalUM system. Your comments will help make this class better. CourseEvalUM is generally open the first couple weeks of December, but this is subject to change by campus.

Right to change information

Although every effort has been made to be complete and accurate, unforeseen circumstances arising during the semester could require the adjustment of any material given here. Consequently, given due notice to students, the instructor reserves the right to change any information on this syllabus or in other course materials.

Accessibility resources: UMD Web Accessibility · UMD Accessibility.