PhD Defense: On the Robustness of Large Lange Model pipelines

Large Language Models (LLMs) models form the core of many modern AI systems. As these systems are increasingly integrated into downstream applications, ensuring the robustness of the entire pipeline from training to deployment has become critical. Real-world deployment exposes these models to a range of vulnerabilities, including data poisoning, model degradation, and performance slowdowns. Addressing these challenges is essential to maintain the reliability, security, and efficiency of LLM-powered systems in production environments. This proposal focuses on addressing certain aspects of the vulnerabilities in these systems.
The first part of the proposal addresses training-time vulnerabilities in the LLM alignment stage, particularly those related to backdoor attacks. In particular it analyzes the possibility of automated stealthy backdoor attacks that can be performed via data poisoning attacks. To this end this part introduces a novel attack method, AdvBDGen, which eliminates the need for manual backdoor design by exploiting the learning capability differences between strong and weak language models.
The second part of the proposal focuses on the robustness of reward models used in the alignment phase. Reward models trained using human preference pairs often experience significant performance degradation on out-of-distribution (OOD) samples, a phenomenon commonly referred to as reward hacking. Prior works have attempted to mitigate this issue through heuristic-based approaches. In contrast, this proposal introduces REFORM, an automated framework that leverages controlled decoding in conjunction with an imperfect reward model to generate class-appropriate OOD samples that expose reward model instabilities.
Finally, the proposal explores the real-world vulnerabilities that LLMs face as reasoning capabilities are integrated into them. In particular, this part of the presentation takes a mechanistic approach to tracing the safety failures of reasoning-induced models back to their base models, thereby improving robustness. Taken together, this defense provides a comprehensive view of the vulnerabilities LLMs face from training through inference.