Differentiable Programming has the potential to impact many different fields in upcoming years. With recent advances in general-purpose GPU (GPGPU), cloud computing, data-driven modeling and deep learning, some of the areas, where differentiable programming has shown success, include computer vision, physics-based simulation, speech analysis to natural language processing. "Differentiable Programming" can be broadly defined as a novel programming paradigm, where programs are treated as compositions of differentiable operations that can then be optimized to fit data -- with tremendous possibility to transform how we can utilize computers to perform calculations on "big data".
This course will examine at how differentiable programming works, from theoretical foundations, practical design and consideration, to system implementation. The course will cover popular concepts like automatic differentiation, models to be integrated with deep learning (including convolutional and recurrent neural networks), as well as some specific instances in differentiable vision, differentiable rendering, differentiable physics, etc. The students are free to choose application of differentiable programming in various topic areas, including Quantum Computing, Genomics, Robotics and AI, Computational Biology and Neuroscience, etc.
The overall goal of this course is to survey some of the most important recent development in differentiable programming, and not to teach you how to train pre-existing models (although you may learn to do that if you choose to do a course project). The aim of this course is to equip you with the broad knowledge and basic skills, so you can understand the literature and implement models and novel differentiable programming concepts. The intent is that after this course, you will develop the ability to critically evaluate and appreciate new system ideas, constructively dissect and criticize, and skillfully identify limitations and potential of various differentiable programming approaches.
Pre-requisite: The predominant focus is on looking at models that can be optimized via gradient-based methods. So, you are expected to have a good grasp of linear (matrix) algebra and matrix calculus, as well as the fundamentals of machine learning, probability and statistics. You should also be comfortable with Python programming and the use of numeric/matrix libraries such as pyTorch, if you choose the 3-credit option (vs. 1-credit for only attending and participation) to do a final course project with significant programming and software components.
================================================================================================================================================================================================================================================================================================================
LECTURES AND APPROXIMATE SCHEDULE
There will be lectures each week. The lectures will all take place either in-person or online, with materials made available through links placed on the course page. By taking part in this course at the minimum of 1-credit option, we expect you to actively participate, attend most lectures, and get involved -- asking thoughtful questions, partaking in stimulating discussion – either in person or via text chat, etc.
A tentative class schedule, illustrating the planned topic and course materials, is posted below. This schedule will evolve as the class progresses throughout the semester. Many of the lectures are coupled with assigned reading. Whenever possible, students are strongly encouraged to read before the lecture takes place. Advanced reading will enhance understanding of the topic, while contributing to much more lively discussion in class. Some of the lectures will be invited guest lectures given by well-known researchers in differentiable programming.
================================================================================================================================================================================================================================================================================================================
COURSE READING MATERIALS
================================================================================================================================================================================================================================================================================================================
ASSIGNMENTS AND PROJECTS
The class grade of each student taking 3-credits is determined by
================================================================================================================================================================================================================================================================================================================
For more information, contact Ming C. Lin, lin@cs.umd.edu.================================================================================================================================================================================================================================================================================================================
Copyright 1999-2023. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the author.
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.