When AR Knows Before You Ask

When digital assistants first appeared on personal computers, they often waited for users to type a command or click a button. As wearable technologies move closer to everyday use, researchers are asking a different question: What if an assistant could offer help at the right moment without being asked and without interrupting? 

At the University of Maryland Department of Computer Science, Ph.D. student Geonsun Lee is studying that shift. Lee is the lead author of “Sensible Agent,” a project developed in collaboration with Google Research that examines how augmented reality assistants can provide support in ways designed to feel less intrusive.

Published at UIST (User Interface Software and Technology) 2025, the research presents Lee’s work on proactive AI agents for augmented reality (AR) systems. Advised by Distinguished University Professor Dinesh Manocha, Lee focuses on developing systems that move beyond traditional voice commands. The prototype explores how an assistant can interpret real-world context, such as where a user is looking or whether their hands are occupied, and adjust how it delivers information.

“We often imagine a future with AR glasses where you can ask questions about your day-to-day life,” Lee said. “But in reality, people may not want to speak out loud all the time, especially in public or when they are repeating the same task.”

Lee said many existing digital assistants are reactive, meaning they respond only after a user issues a command. The Sensible Agent project instead studies proactive agents that anticipate needs based on context. However, she noted that proactive systems risk becoming disruptive if not carefully designed.

“There is a long-standing concern in human-computer interaction about intrusive assistants,” Lee said, referring to early software tools that frequently interrupted users. “If an agent asks too often or at the wrong time, it can become a burden.”

To address that concern, the project examines both how an AR agent presents information and how users respond to it. On the agent side, researchers explored different query formats. In some situations, a system might display a simple yes-or-no prompt. In others, it could present multiple options or a minimal visual icon that signals an available action.

Lee described a cooking scenario as one example. If a user routinely prepares the same recipe, the agent may display a small timer icon rather than provide a full spoken suggestion. If the user is unfamiliar with the ingredients, the system could instead offer recipe recommendations or suggest additional items to purchase.

Presentation modality was another focus. Depending on the context, the agent might use a visual panel, an audio cue or both. The goal, Lee said, was to align the interaction with the user’s sensory availability, such as whether their gaze, hands or hearing were already engaged.

To evaluate these design choices, the team conducted a preliminary user study with 10 participants, comparing the proactive prototype with a more traditional reactive system. 

“Even though there were trade-offs, the participants reported that the interaction felt closer to how they communicate with other people,” Lee said. “They did not have to list everything explicitly in a command.”

Working with Google, she said, positively impacted the work and provided access to advanced computational resources and researchers developing large AI models.

“It was helpful to discuss design decisions directly with people building these systems,” Lee said.

Her internship manager, alum Ruofei Du (Ph.D. ’18, computer science), an interactive perception and graphics lead at Google XR, is also involved in an open-source initiative called XR Blocks. The project aims to facilitate the development of immersive experiences using web-based tools. Lee said the Sensible Agent framework could eventually serve as a test template within that ecosystem.

Looking ahead, Lee said context-aware interaction design is one step toward making AR assistants more practical in educational, professional and daily settings. Rather than embedding a chatbot inside glasses, she said future systems may need to interpret surroundings more carefully and respond in ways that align with human routines.

“As researchers, we still have many design questions to answer,” Lee said. “But understanding how to interact naturally with these agents is an important part of that process.”

—Story by Samuel Malede Zewdu, CS Communications