Benchmarking Coding Agents on Underspecified Tasks
Coding agents often proceed on underspecified requests instead of asking for clarification. From 4,677 real coding-agent sessions, we built 171 sandboxed tasks with executable verifiers and a simulated user who reveals the missing specification only when asked, and we score each question by its cost of not asking.
Key result: Fable 5.1 asked in only 18% of runs, and 60% of its questions were low-value (vs. an 80% base rate).
Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
RL-trained agents can learn to call tools in response to irrelevant prompt cues. We use counterfactual evaluations to isolate these shortcuts and introduce a tool-necessity reward to discourage unnecessary calls.
Key result: On GSM8K with the WEB_INDEX cue present, unnecessary search calls fall from 40.1% to 0.2% after the reward intervention, while answer accuracy improves.
Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?
Vision-language models can latch onto misleading visual cues. We curated 1,364 multiple-choice VQA questions across 124 spurious correlation types, drawn from real model errors and synthetic counterfactuals, to test whether models generalize beyond correlations they have seen.
Key result: Fine-tuning Qwen2.5-VL-7B on diverse synthetic examples raised accuracy on held-out spurious patterns from 35.2% to 78.4%.
Label-Efficient Group Robustness via Out-of-Distribution Concept Curation
Fixing spurious correlations usually requires costly per-example group labels. Concept DRO instead curates out-of-distribution concept images, including AI-generated ones, to estimate group labels and train distributionally robust models.
Key result: Up to 33.1% improvement over baselines without any explicit spurious attribute annotations.