PhD Defense: Foundation Models as Priors for Embodied, Language-Guided Navigation

Over the past decade, the use of machine learning in robotics has evolved from specialized models tailored for individual tasks towards robot-data-driven finetuning of Foundation Models (FMs), i.e., large deep learning models that exhibit generalizable, open-ended reasoning capabilities. Finetuning FMs for robotics however requires on-device robot data that can be painstakingly hard to collect and clean. In unstructured, human-populated environments like homes and office spaces, such data gathering is further complicated by safety and privacy concerns.
We present in this dissertation, methods to use FMs trained on passive, non-robot data (like Vision-Language Models (VLMs) and Large-Language Models (LLMs)) for language-guided navigation tasks in unstructured, human-populated environments. Rather than fine-tuning FMs using robot-specific data, our methods favour the use of FMs off-the-shelf as priors for open-ended commonsense reasoning, enabling a practical, modular solution for robot deployment in human-populated environments.
In Part I, we introduce two methods that use VLMs and LLMs for language-guided navigation tasks in household environments. The first method, CLIP-Nav develops a VLM-based approach for the task of Vision-and-Language Navigation (VLN). Evaluating CLIP-Nav on the Matterport3D environment outperforms even a fully supervised baseline by ~2% while improving generalizability by 17% when measured by Relative Change in Success. Next, we present Language-Guided Exploration (LGX) that uses an LLM as a commonsense prior to solve the ObjectNav task in the RoboTHOR environment, achieving a 27% improvement over contemporary methods without robot-specific training. Both these methods establish zero-shot benchmarks for language-guided navigation tasks.
In Part II, we place these FM-based agents in practical deployment settings with access to an indoor surveillance camera, or priors on human object-placement habits. We then present methods for the agent to leverage the external information from such situations to improve its performance, without having to finetune. In the first work, by allowing the agent to communicate with an overhead surveillance camera, we improve CLIP-Nav's object-finding success by ~10% in both simulation and real-world Turtlebot experiments. In the second work, we introduce an algorithm to model and learn human object-placement habits, and incorporating this information into LGX improves its non-stationary object-finding success by nearly 10%.
Finally, in Part III, we present three methods to support the practical zero-shot deployment of embodied navigation agents. First, we introduce an LLM-based tool to synthesize human-like wayfinding instructions in unseen environments, which is useful for data augmentation in-the-wild. Second, we develop an algorithm that generates queries about situational context, such as `Is the house ready for sleeptime?', which can be used to proactively anticipate user queries after deployment. Lastly, we propose a novel memory architecture that adds a trainable memory head to an FM backbone to actively filter visual observations. This work improves embodied task performance for low-parameter FMs by ~6% while reducing memory footprint.