PhD Proposal: A Programming Language for Human Action

As humans begin to exist in a world of networks and machines, it becomes essential to understand human action. Indeed, the last several years have seen an unprecedented development of techniques in several disciplines for capturing, sensing, modeling, recognizing, replicating and predicting human behavior. For example, in the field of Computer Vision, several models have been proposed for modeling human activity for the purpose of surveillance or semantic indexing into databases. In the field of Robotics, an abundance of models have been proposed for the purpose of imitation learning (Programming by Demonstration). Despite this development, there is little (if any) synergy among the different disciplines. To model human action, different research groups use different frameworks; dynamical systems, Hidden Markov models, Conditional Random Fields, grammars and ad-hoc finite automata are only some of formalizations that have been employed to model human action. While fundamentally very similar from a theoretical point of view, each of these models is built with a very specific application in mind. For example, a discriminative model trained to visually recognize certain actions might be of little use in a Robotic setting. We believe that thinking of human activity as a formal program provides the means to unifying approaches to action modeling in a solid and foundational manner. We propose to define a simple and intuitive language named AL (Action Language), whose syntax will be very similar to that of currently popular procedural programming languages and whose semantics will model exactly the way AL code execution would interact with the state of the real world (i.e. effects of manipulations involving real objects). We also propose to implement 2 systems: VALC (Visual AL Compiler), that will automatically translate visual observations of manipulation actions to AL programs, and VALD (Visual AL Debugger), an Augmented Reality system that, given an AL program, will guide a user through its execution by providing real-time visual instructions as well as corrective feedback upon program violations.
Examining Committee:
Chair: Dr. Yiannis Aloimonos
Dept. rep: Dr. Nick Roussopoulos
Member: Dr. Cornelia Fermuller