PhD Proposal: Robot Learning from Physical Interactions: Perceiving, Acting and Learning from Contact Force

Robots learn about the physical world through interaction, yet most robot-learning data captures only visual effects rather than the forces that define those interactions. This thesis argues that learning from physical interaction requires three capabilities: perceiving interaction from vision, measuring force where it is generated, and acting to acquire new physical knowledge.
Our dataset FEEL addresses perception by pairing roughly 3 million egocentric frames with synchronized force, enabling force-supervised contact understanding and action representation learning. Out hardware-software solution ForceBand addresses scalable measurement using wrist sEMG to estimate per-finger forces, reducing error relative to vision-based estimators and enabling force-aware robot policies learned from human video. Out interactive perception and planning system Interactive-FAR addresses action by using force feedback during pushing to update object affordances and improve navigation efficiency in simulation.
The proposed work scales these ideas further by learning contact from large human datasets using mesh geometry and visually restored tactile-glove data, and by enabling robots to autonomously collect contact-rich experience through play and learn world models for dexterous in-hand reorientation.