PhD Proposal: From Form to Function: Detecting the Affordance of Tool Parts Using Geometric Features

With recent advances in robotics, general purpose robots like Baxter are quickly becoming a reality. As robots begin to collaborate with humans in everyday workspaces, they will need to understand the functions of objects and their parts. To cut an apple or hammer a nail, robots need to not just know a tool's name, but they must find its parts and identify their potential functions, or affordances. As Gibson remarked, "If you know what can be done with a[n] object, what it can be used for, you can call it whatever you please."
Intuitively, the geometry of a part is closely related to its affordance, so in our preliminary work we investigated how the affordances of tool parts can be predicted using geometric features from RGB-D sensors like Kinect. We developed several approaches to learn affordance from geometric features: using superpixel based hierarchical sparse coding, structured random forests, and convolutional neural networks. To evaluate the proposed methods, we constructed a large RGB-D dataset where parts are labeled with multiple affordances. Experiments over sequences containing clutter, occlusions, and viewpoint changes showed that the approaches provide precise predictions that can be used by a robot.
These preliminary results showed that geometric features are key to predicting affordance. Following these insights, we propose to develop a system which uses geometric features to predict object affordance in videos of complex scenes. We will investigate how different temporal and geometric features can be used to predict affordance, and how mid-level features can be integrated into the learning framework. This will allow robots to predict the potential functions of tool parts and their effects, providing a basis for higher level reasoning about objects and actions.
Examining Committee:
Committee Chair: - Dr. Yiannis Aloimonos
Dept's Representative - Dr. Hal Daume III
Committee Member(s): - Dr. David Jacobs
- Dr. Cornelia Fermuller