PhD Defense: ENABLING SAFE ROBOTIC COMPANIONS THROUGH HUMAN-AWARE SIMULATION, NATURAL COMMUNICATION, AND EMERGENCY RESPONSE

Robotic companions have the potential to revolutionize daily life by helping users with everyday tasks, improving safety, and enhancing quality of life in home environments. However, advancing these robotic companions requires addressing three distinct technical challenges. First, representative household simulation environments lack realistic human agents that move and interact naturally with scenes, which limits the development and testing of human-robot interaction algorithms. Second, robots must be able to interpret and respond to user commands given in natural language, a task complicated by the inherent ambiguity in human communication. Finally, to meaningfully improve home safety, robots must be able to detect and respond to emergency situations. This dissertation addresses each of these challenges through novel methods spanning simulation, communication, and emergency response.
In simulation, we present PAAK and PACE, novel methods for creating realistic virtual human agents in 3D environments. PAAK places motion-captured human animations into 3Dscenes, maintaining human-scene interactions by using “keyframes”—the most important frames for modeling those interactions. PACE builds upon this by modifying motion sequences to adapt to dense, cluttered environments, adjusting the high-DOF pose at each frame to account for unique geometric constraints, and received Best Paper Honorable Mention at IEEE VR 2023. For user communication, we present LGX and LBAP, which enable robots to understand natural language commands and handle ambiguity. LGX leverages Large Language Models (LLMs) for language-guided zero-shot object navigation, achieving a 27% improvement in zero-shot success rate over prior methods on RoboTHOR, with successful real-world validation on a TurtleBot 2 platform. LBAP uses Bayesian inference for uncertainty alignment in LLM planners, mitigating hallucinations and better aligning confidence measures with probability of success, reducing human intervention by over 33% at 70% success rate in real-world experiments on a ClearPath TurtleBot.
For household emergencies, we present SafetyDetect and HomeEmergency, enabling robots to detect and respond to dangerous situations. SafetyDetect introduces a dataset of 1000 anomalous home scenarios and an LLM-based approach using scene graphs to identify unsafe or unsanitary conditions, achieving a 96% anomaly detection rate in simulation and 89% on a ClearPath TurtleBot in real-world scenarios. HomeEmergency addresses actively occurring emergencies like falls and fires using audio localization through a novel probabilistic dynamic scene graph, where Bayesian inference enables efficient agent localization, with validation across 30 real-world experiments in apartment environments. While PAAK is primarily a simulation tool, our other methods include real-world validation: LGX, LBAP, SafetyDetect, and HomeEmergency all demonstrate effectiveness on physical robotic platforms. Together, these contributions advance robotic companions across research tools, communication capabilities, and safety features, moving toward more helpful and trustworthy robotic companions.