Teaching AI How to Speak

Asking someone to slow down, speak louder or wait until a sentence is finished is a routine part of conversation. But when the listener is an artificial intelligence voice assistant, those simple requests may not produce the expected response. For alum Utkarsh Tyagi (M.S. ’25, computer science), that limitation raised a research question: Can AI systems learn not only what to say, but how and when to say it?

Tyagi, advised by Distinguished University Professor Dinesh Manocha, led the development of SteerDuplex, a speech-to-speech AI model designed to adjust its speaking behavior in response to instructions while maintaining real-time conversations.

The project, conducted with researchers from Scale AI and UMD's GAMMA Lab, also introduces SteerBench, a benchmark for evaluating how well voice-based AI systems respond to instructions about tone, speaking speed, persona and vocal style.

“There are multiple signals to look out for, like the turn-taking, the back-channeling, background noise and the pause handling,” Tyagi said. “What might sound natural to me might not sound natural to anyone else.”

Closing the Gap

Tyagi's interest in conversational AI grew from the differences he observed between text-based systems and spoken interactions. While testing existing open-source speech models, he found that many struggled to follow instructions about how they should speak. For example, a request to slow down or increase speaking volume might have little effect.

This observation motivated him to investigate steerability: an AI system's ability to adjust its conversational behavior in response to a user's instructions without disrupting the exchange.

To address the problem, Tyagi and his collaborators developed SteerDuplex using Moshi, an existing open-source speech model. They used supervised training to introduce new capabilities, followed by a two-stage reinforcement learning process to improve conversational timing and responses to interruptions.

“The problem was when we introduce a lot of new capabilities into the model, it tends to forget what it already had,” Tyagi said. “So we had to make sure that it does not lose any existing capabilities while also having the new capabilities.”

One challenge involved designing training rewards that encouraged appropriate responses without introducing unintended behaviors. For example, rewarding a system for yielding to interruptions could cause it to stop speaking altogether.

To measure the model's performance, the team developed SteerBench, which includes 390 spoken prompts and 1,067 human-written evaluation criteria. The researchers also introduced a method that allows an audio language model to compare generated speech with reference recordings rather than relying entirely on human evaluators.

“You give it an audio which is already talking slowly with the same content, and then ask it to compare rather than judge,” Tyagi said. “Rather than asking it an open-ended question, we ask it to compare from a reference audio which is already talking slowly.”

According to the research paper, supervised training improved the model's average pass rate on audio-steering tasks by 44.5 percentage points compared with the strongest open-source baseline evaluated. Reinforcement learning also improved interruption handling and reduced instances in which the model began speaking during user pauses.

UMD's Research Scene

At UMD, Tyagi worked under Manocha and Professor Ramani Duraiswami, gaining experience in artificial intelligence research and contributing to academic publications. He credits that experience with helping him transition from software engineering into research.

“UMD was transformational for my career,” he said. “I was able to gain knowledge and experience in research areas that were entirely new to me.”

That connection has continued beyond graduation. Tyagi regularly collaborates with Manocha and GAMMA Lab researchers on conversational AI projects. 

“Even though I'm not their student, they love to collaborate with me and contribute to my papers,” Tyagi said. “I'm very much in touch with them and love to work with them still.”

Beyond the Lab

Tyagi sees potential applications for SteerDuplex in personal assistants, customer service systems and other settings where people interact with AI through speech. More responsive voice systems could allow users to complete everyday tasks through conversation rather than repeatedly typing instructions.

He also identified possible applications in mental health support, while emphasizing that AI systems should not replace therapists.

“SteerDuplex could enable more adaptable voice assistants, tutors, and conversational characters that follow requests about tone, persona, accent, and speaking pace while handling interruptions naturally,” Manocha said. “Its novelty is combining vocal steerability with full-duplex interaction training, alongside SteerBench, which evaluates both response content and audible delivery.”

Although SteerDuplex demonstrated improvements in conversational control, Tyagi acknowledged that the model still faces limitations in reasoning and handling complex requests. His next research goal is to develop systems that can process information more extensively while maintaining the responsiveness required for real-time conversations.

Beyond improving individual models, Tyagi sees SteerBench as a way for researchers to measure conversational control and track progress as new speech systems are developed. He hopes the work will encourage further research into making AI more adaptable to the changing demands of human conversation.

“Having more steerable models makes them more human,” Tyagi said. “I think what the benchmark and the model do is introduce a metric which future models can use to track their own capabilities and ultimately lead to models being much more natural to talk with.”

—Story by Samuel Malede Zewdu, CS Communications