Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
Large vision-language models (LVLMs) like GPT-4V, Gemini 1.5, Claude, and Qwen 2.5-VL demonstrate impressive performance across various tasks, but can they truly generalize beyond spurious correlations learned during training? This research systematically investigates whether state-of-the-art vision-language models can overcome spurious associations when confronted with scenarios where previously learned patterns break down or are inverted.
We create a comprehensive evaluation framework to assess model robustness when spurious correlations are disrupted. Our findings reveal critical limitations: even advanced models struggle to learn robust features independent of spurious correlations, highlighting fundamental challenges in developing truly reliable AI systems that can generalize to distribution shifts.
Key Contributions: (1) Documented how vision-language models develop and rely on spurious correlations, (2) Created systematic benchmarks for evaluating robustness to spurious pattern disruptions, (3) Evaluated multiple state-of-the-art LVLMs to determine their vulnerability to distribution shifts involving spurious correlations.
Label-Efficient Group Robustness via Out-of-Distribution Concept Curation
Building machine learning models that maintain consistent performance across different demographic groups typically requires extensive labeled datasets—a costly and time-consuming process. This research introduces a novel approach that achieves group robustness while dramatically reducing annotation requirements through strategic use of out-of-distribution concepts.
Rather than relying on exhaustive labeling, we demonstrate how carefully curated concepts that fall outside the training distribution can improve model fairness across demographic groups. By incorporating these strategically selected concepts during training, models learn more robust feature representations that generalize better across diverse populations.
Key Contributions: (1) A novel methodology for achieving group robustness through out-of-distribution concept curation, (2) Demonstrated label efficiency that significantly reduces annotation requirements while maintaining or exceeding fully supervised performance, (3) A practical framework for practitioners developing equitable AI systems with constrained labeling budgets.