PhD Defense: Mechanistic and Functional Interpretability for Attribution in Neural Networks
IRB-4107 https://umd.zoom.us/j/9500203024
With the introduction of larger and more powerful neural networks each year, their inner workings become increasingly opaque. The attribution problem asks which parts of the input or model are responsible for a given model output or behavior. Many interpretability problems, from discovering semantically meaningful directions in representation space to citing the sources behind a generated answer, can be understood as instances of this problem. My research approaches it from both mechanistic and functional perspectives.
From a mechanistic perspective, I introduce RepDecompose, a decomposition-based technique for input and model attribution in ViTs and ViT-CNN hybrids, and HARP, a training-free interpretability agent that discovers concepts in LLM activations and rivals expensive training-based methods such as sparse autoencoders.
From a functional perspective, treating the model as a function of its input, I show that chain-of-thought explanations in multimodal LLMs are unreliable for input attribution. Finally, I introduce DecompTune, which trains LLMs to decompose answers and attribute them to source documents, matching frontier models.