PhD Proposal: Fast personalization of language models via personalized preference modeling and hyper-adaptation

Personalization of language models aims to align model responses with the unique preferences of a given user. For example, a model response containing medical jargon is appropriate for a medical doctor with years of experience but inappropriate for a young child. Most prior work on personalization has proposed personalizing language models by fine-tuning the language model for each user. However, fine-tuning is costly, and is thus prohibitive for large user bases. As language models proliferate, scaling to potentially billions of users, the need for fast and efficient personalization becomes a central problem. Our primary goal is to develop datasets and methods for personalizing language models that yield the personalized model not by slow operations (such as fine-tuning for each user), but by fast operations (such as predicting the parameters to an adaptation in one forward pass). Hence, we call this problem fast personalization, which is relatively underexplored in existing work.
We propose to develop a principled approach to fast personalization by extending the classical Bradley--Terry preference model to the personalized setting through a user-specific offset δ(x,y,u) to the universal reward r*(x,y). Our personalized preference model serves as the foundation for our fast personalization methods, which adapt the language model in a single forward pass. We propose three research directions. First, we model δ(x,y,u) using a multi-dimensional generalization of item response theory to learn interpretable latent features of users, prompts, and responses. Second, we mathematically derive a personalized variant of Direct Preference Optimization (DPO) from our personalized preference model, yield a principled objective for training a hypernetwork to generate user-specific adaptation parameters in one forward pass. Third, we introduce hyper-steering, in which we infer steering directions in the model's activation space corresponding each each learned latent feature of model responses, and train a hypernetwork to map user data to steering vector coefficients that modulate the models' activations in one forward pass at inference time. We also incorporate insights from our prior work on fast personalization via low-rank hypertuning, which provides a preliminary investigation of one-pass multi-layer adaptation.
Together, these contributions will provide (1) a framework for discovering latent variables of users, prompts, and responses, (2) a principled objective for personalization with hypernetworks, and (3) a novel method for fast personalization via hyper-steering, enabling fast, scalable personalization of language models.