PhD Defense: Selecting, Generating, and Targeting: Gradient-Based Pretraining Data Curation at Scale

The quality and composition of training data now shape much of what modern machine learning can do. Our work is unified by one recurring question: which data matters, and how do we identify or act on it efficiently? This talk traces that question across four lines of work, converging on gradient-based data curation for large language models.We begin with active learning: which examples should we annotate under a fixed budget? Popular methods fail to beat random sampling at ImageNet scale, where class imbalance makes them over-query a few classes and starve others. Our method, Balanced Selection, enforces an even class distribution near decision boundaries, and matches state-of-the-art ImageNet accuracy using only 71% of labels. The lesson: the composition of the selected set drives performance, not just per-example utility.We next turn to federated learning, where the gradient becomes an object to shape and search. Users share only gradients, and prior attacks fail under large-scale secure aggregation. We introduce the first attack that stays effective under arbitrarily large aggregation. A malicious server reprograms attention heads to tag tokens near a chosen keyword, then filters out the rest, isolating the small slice of data that matters.In generative modeling, we study flexible, controllable generation. We first show that diffusion does not require Gaussian noise: a whole family of models can be built from arbitrary deterministic degradations. We then show that any off-the-shelf network can steer a fixed diffusion model without retraining. We later repurpose the steering technique of a frozen generator to generate training data.These threads converge in our recent work on gradient-based data selection for LLM pretraining. There, the inner product between a document's gradient and a reference example estimates how much training on it lowers the reference loss. Two projects make this practical at scale. The first attacks its cost, computing these inner products implicitly during the backward pass with custom kernels up to 12x faster than a dense matrix multiply. Our centerpiece, Benchmax, attacks fidelity: it scores each reference example individually instead of averaging the benchmark into one gradient, then sketches the gradients and distills them into a fast classifier. This pipeline scores about 1T tokens for roughly $500 and beats strong DCLM and fastText baselines.Building on this foundation, we propose to push the gradient signal further. First, we generate synthetic data by training a document rewriter with reinforcement learning against a gradient-alignment reward. Second, we target the task-adapted model directly by an approximation where a directional change between base model and fine-tuned model is added to the current model. Together these form a unified program for selecting, generating, and targeting pretraining data at scale.