The Three P’s of Modern Computer Vision: Pixels, Perception and Physics
4105
For decades, computer vision has been guided by what Malik et al., called the Three R’s: Recognition, Reconstruction and Reorganization. We have made remarkable progress on all three and some would even argue they are nearly solved. Many things we struggled to do a few years ago can now be done with an off-the-shelf model. So, what’s next for computer vision? Do we all become roboticists, as many in our community are doing or is there a broader direction to explore?
In this talk, I will argue that there is. Our progress on these perception problems—the Three R’s—gives us a foundation for asking what happens when we change the world we see. I will describe our group’s research around what I call the Three P’s: Pixels, Perception and Physics.
I will begin with how 3D structure lets us control camera and object motion in generated videos. But how do we tell whether a convincing video gets the physics right? I will introduce a controlled benchmark for evaluating video models, then explore how simulation and learned video generation can predict how objects move, deform, and tear under applied forces. I will conclude by looking beyond vision by introducing OpenTac, our new tactile skin that senses normal and shear forces and can be customized to fit different surfaces. I will discuss how touch can complement vision in robotic interaction, with potential applications in understanding material properties.