PhD Proposal: Rethinking Question Answering Evaluation: Human-Centered Benchmarks for Reasoning, Learning, and Multimodal Understanding

The availability of large-scale datasets and powerful language models has transformed open-domain question answering (QA). Despite these advances, existing QA benchmarks, such as Natural Questions (NQ), remain limited in scope, often suffering from data sparsity, temporal obsolescence, under-specified queries, and evaluation methodologies that inadequately capture differences between human and machine reasoning. Existing QA benchmarks, such as Natural Questions (NQ), remain limited in scope, often suffering from data sparsity, temporal obsolescence, under-specified queries, and evaluation methodologies that inadequately distinguish human reasoning from machine performance. This dissertation explores the design of richer question-answering resources that improve both model training and evaluation across modalities. Specifically, it introduces (1) naturalization, a transformation pipeline that converts structured trivia questions into naturalistic short-form QA pairs, and (2) AudioQA, a multimodal benchmark grounded in real-world audio and pyramidal question construction. Together, these contributions provide new approaches for creating more realistic, informative, and human-centered QA datasets. The naturalization pipeline focuses on transforming rich, multi-sentence trivia questions—typically found in QuizBowl tournaments—into standalone, factoid-style queries that match the linguistic structure and format of NQ. The transformation process includes syntactic simplification, clause extraction, and preserving semantic accuracy while aligning with QA model expectations. Models trained with naturalization-transformed data achieve strong zero-shot and supervised performance on the NQ benchmark, often approaching or exceeding baselines trained on the original NQ dataset. AudioQA represents a complementary advance, introducing a human-authored audio QA benchmark composed of over 18,000 segmented clips from quiz tournaments, music databases, and trivia archives. Each clip is annotated with QA prompts and metadata, supporting fine-grained evaluation of auditory comprehension tasks. Current state-of-the-art multimodal models—including AudioGPT, MU-LLaMA, and Flamingo—struggle with AudioQA, while human participants maintain high accuracy. This discrepancy underscores the need for more robust auditory reasoning capabilities in LLM-based architectures. In addition to introducing these resources, the dissertation evaluates the impact of dataset combinations across various downstream tasks. Experiments reveal that mixing NQ with naturalization transformed data improves factual recall and interpretability, while combining text with AudioQA enhances resilience to disfluency and improves contextual comprehension in audio-rich environments. Transforming other trivia datasets (e.g., Jeopardy, TriviaQA, AI King) into NQ-style questions boosts model performance on the MMLU benchmark, a proxy for general-purpose reasoning. These findings support the hypothesis that different QA tasks benefit from specific combinations of structured, naturalized, and multimodal training data.
This proposal concludes by presenting a research agenda centered on multimodal adversarial pyramidal question answering. Building on the datasets and methodologies developed in this dissertation, future work will focus on the creation of Earudite, a platform for collecting and evaluating pyramidal audio and visual questions, the development of adversarial evaluation frameworks that directly compare human and AI performance, and the use of Item Response Theory to model question difficulty and participant expertise. The long-term goal is to establish a unified framework for measuring human and machine reasoning across modalities. Ultimately, this dissertation contributes both practical resources and principled evaluation methodologies for building QA systems that better reflect human knowledge and understanding.