Groundwork for Open Speech Foundation Models: Data, Recipes, and Task-Specific Models

Building speech foundation models is no longer the exclusive domain of large companies, but reproducing one still takes more than a published architecture. The hard part is the groundwork: collecting, cleaning, and automatically annotating data before training begins. Large-scale audio collections are abundant, yet raw audio and transcripts are often misaligned, mislabeled, or otherwise unusable without careful processing. This talk uses our fully transparent open Whisper-style speech models (OWSM) as a case study and focuses on the groundwork behind them. I show how OWSM v4 narrowed a web-scale pool of multilingual speech, around 500K hours from YODAS v2, to a usable training set through alignment, language identification, and CTC-based scoring. I also show how the same kind of pipeline carries over to YODAS v3, a larger, more multilingual release recorded at higher fidelity. I argue that task-specific models are not a stepping stone to be replaced by foundation models; they are the tools that make the data usable. Finally, I present the Phonetic Open Whisper-Style Speech Foundation Model (POWSM), which applies the same recipe to phone recognition, showing that task-specific linguistic and phonetic tools are the key to the groundwork. I then briefly discuss unified speech models spanning generation and understanding as a direction that follows the same open-science principles.