PhD Defense: Advancing Audio Intelligence for Perception, Reasoning, & Generation

Talk
Sonal Kumar
Time: 
12.01.2026 11:00 to 12:30
Location: 

Audio, spanning speech, environmental sound, and music, is one of the primary channels through which humans perceive the world, yet it remains underexplored in AI relative to language and vision. Large Language Models have enabled open-ended audio question answering and text-to-audio generation, but as separate systems with clear blind spots: audio-language models often answer from language priors rather than the signal, cannot say when or where sounds occur, and assume a single input clip; benchmarks reward shallow recognition; and generators offer only coarse control and share no representation with the models that understand audio. This dissertation develops models, data, and benchmarks that address these gaps across perception, reasoning, and generation, and then unifies them.
On the perception side, CompA teaches audio-language models event order and attribute binding through composition-aware benchmarks and a contrastive objective. TAC generates timestamped captions of dense, overlapping audio with a low hallucination rate and serves as a grounded interface to text-only reasoners. SEA-LM brings egocentric spatial understanding to smart-glasses microphone arrays, telling where sounds come from and whether the wearer or a bystander is speaking. Building on grounded perception, GAMA combines complementary audio representations with an LLM and synthetic complex-reasoning instructions, and PolyAudio extends reasoning from one clip to many without sacrificing single-clip skill.
To measure progress, MMAU and MMAU-Pro evaluate expert-level reasoning across speech, sound, and music, including long-form, multi-audio, and spatial settings, and reveal a large gap between models and humans. Audio Hallucination Attacks show that accuracy overestimates grounding: implicit questions and injected speech lead every evaluated model to describe absent sounds, and preference training on targeted data reduces this failure. On the generation side, Synthio uses preference-aligned text-to-audio synthesis to augment low-resource classification, SILA adds fine-grained acoustic control to text-to-audio models, and ProSE guides speech enhancement with a compact diffusion prior.
Finally, UniGAMA brings these threads together in a single model for audio reasoning, generation, and text-guided editing. A language-bridged refiner, trained jointly with the language backbone, lets generation supervision shape the representations used for understanding, and at fixed capacity joint training outperforms single-task counterparts. Together, these contributions move audio intelligence toward systems whose perception is grounded in the signal, whose reasoning operates over structured auditory evidence, and whose generation is controllable and shares its representations with understanding.