New UMD Research Brings Smarter, More Subtle Assistance to AR Glasses

Recent innovations, such as Google's Project Astra, exemplify the potential of proactive agents embedded in augmented reality (AR) glasses to offer intelligent assistance that anticipates user needs and seamlessly integrates into everyday life. These agents promise remarkable convenience, from effortlessly navigating unfamiliar transit hubs to discreetly offering timely suggestions in crowded spaces. Yet, today’s agents remain constrained by a significant limitation: they predominantly rely on explicit verbal commands from users. This requirement can be awkward or disruptive in social environments, cognitively taxing in time-sensitive scenarios, or simply impractical.

To address these challenges, we introduce Sensible Agent, published at UIST 2025, a framework designed for unobtrusive interaction with proactive AR agents. Led by Ruofei Du, Interactive Perception & Graphics Lead at Google, and Geonsun Lee, a University of Maryland Department of Computer Science Ph.D. student and Google XR Student Researcher advised by Distinguished University Professor Dinesh Manocha, the project represents a collaboration bridging cutting-edge academic research and industry innovation.

Sensible Agent builds on the team’s prior research in Human I/O and fundamentally reshapes interaction by anticipating user intentions and determining the best approach to deliver assistance. It leverages real-time multimodal context sensing, subtle gestures, gaze input, and minimal visual cues to offer unobtrusive, contextually appropriate assistance—marking a crucial step toward truly integrated, socially aware AR systems that respect user context, minimize cognitive disruption, and make proactive digital assistance practical for daily life.

Click HERE to read the full article