PhD Defense: Understanding and Exploiting What Neural Networks Learn: Unlearnable Datasets and Inactive Attention Heads

Neural networks have become the dominant technique across domains due to their ability to find statistical patterns that generalize beyond their training data. However, they also exhibit learning biases and learned behaviors that can be understood and exploited. This dissertation investigates two complementary questions: what patterns neural networks preferentially learn during training, and what computations they perform unnecessarily during inference. We study the first question in the image domain, using primarily convolutional neural networks. We study the second question in the text domain, using pretrained transformer language models. We develop methods that both manipulate learning behavior and identify new insights into transformer computation.
The first part of this dissertation focuses on dataset security through unlearnable datasets. We study why certain perturbations prevent generalization, showing that perturbations which are learned quickly by convolutional neural networks are most effective at causing overfitting. Building on this insight, we introduce autoregressive perturbations, a model-free and backpropagation-free method for generating reusable unlearnable datasets that remain effective against common defenses. We then examine the limitations of perturbation-based data protection, demonstrating that neural networks can still learn useful representations from ostensibly unlearnable data. We develop methods based on orthogonal projection and image sharpening that recover performance on both bounded and highly corrupted unlearnable datasets, and further show that JPEG compression similarly defeats protections against AI image editing. Together, these results challenge the long-term viability of imperceptible perturbation-based approaches to dataset security.
The second part of this dissertation investigates inefficiencies in transformer language models. We study the attention sink phenomenon and develop score functions for identifying inactive attention heads. By ablating attention heads, our analysis reveals that substantially more attention heads are inactive than previously estimated, allowing more than 12% of attention heads to be zeroed during inference while maintaining model performance. These findings suggest opportunities for reducing transformer inference costs and motivate future architectures that dynamically allocate attention computation. Collectively, we demonstrate that understanding inductive biases and computational structure learned by neural networks provides a foundation for both attacking and improving modern neural networks.