PhD Defense: Toward Reliable Supervision for Foundation Models

Foundation models have made rapid progress in reasoning, alignment, and multimodal understanding, but their reliability critically depends on the quality of supervision used during learning. Such supervision may come from learned reward models, verifiable outcomes, expert models, the model itself, or cross-modal alignment. Each source is useful but limited: learned rewards can be noisy, verifiers are reliable but sparse, expert judges may be unavailable, weak supervisors can be biased, and visual evidence can be ignored by models that over-rely on textual priors. This dissertation studies how to improve large models when supervision is sparse, imperfect, indirect, or distributed across models and modalities.
First, we study process supervision for Large Reasoning Models (LRMs). Process Reward Models (PRMs) provide step-level feedback, but we show that they can overcredit plausible yet incorrect reasoning steps, producing false positives that mislead search and policy optimization. We introduce PRISM, a policy-aware PRM training framework based on contrastive step-level comparisons, temporal-lookahead hard negatives, and a difficulty-aware curriculum. We then examine how these improved but still imperfect process rewards can be used safely during policy refinement. In VeriGate, we integrate PRM feedback into Group Relative Policy Optimization (GRPO) while preserving verifier authority: verifier rewards drive learning when they are informative, and PRM-based dense supervision is used only when verifier rewards are degenerate. Step scores are converted into future-cumulated token-level advantages, reducing reward-hacking artifacts.
Second, we investigate improvement without external reward models, verifiers, or expert judges. Uncertainty-Aware Answer Selection calibrates token log-likelihoods across diverse Large Language Models (LLMs) to select the most reliable response from multiple candidates, enabling lightweight multi-LLM and Best-of-N selection using only model confidence. EnsemW2S extends this idea to training-time weak-to-strong (W2S) supervision: weak experts are refined with limited labeled data, combined at the token level, and used to generate pseudo-labels for a stronger student. This improves generalization across in-distribution, out-of-distribution, and high-difficulty tasks.
Finally, we extend this supervision perspective to Large Vision-Language Models (LVLMs). These models often hallucinate because their language backbones over-rely on textual priors and underuse visual evidence. VisAlign refines textual embeddings with global visual context, improving representation-level alignment between language and vision and reducing hallucinations across multimodal benchmarks.