Video Intelligence for Understanding Behavior and Guiding Action

Modern video-language models have become remarkably good at describing what is happening in a video. Yet their ability to understand the behavior they observe, and to act on that understanding, remains limited. This talk presents our group's work on building these missing capabilities. I will begin with foundational methods for scalable video perception, from space-time attention to hierarchical representations and active search over long videos. I will then introduce strategic video intelligence, using team sports as a natural microworld in which cooperative and competitive behavior can be studied rigorously, and present two complementary lines of work within it. The first investigates whether video generation models can simulate strategic multi-agent behavior well enough to support better decisions. The second examines how agents can study behavioral histories across large multimodal corpora, compiling repeated analytical work into reusable representations, abstractions, and procedures. Finally, I will present WatchAct, a benchmark for robot manipulation grounded in observed human behavior, where a robot watches a person and then has to act on what it saw. Together, these works progress from perceiving behavior, to understanding it, to guiding action.