The right answer can hide the wrong action
An agent can answer correctly while calling tools it never needed. We study how reinforcement learning turns irrelevant prompt cues into tool-use shortcuts, and introduce a reward that discourages unnecessary calls.
GSM8K with the WEB_INDEX cue present. Results compare cue-trained agents before and after the reward intervention; see Table 1 of the paper.
How we isolate the shortcut
Train with a correlation
Pair factual questions with search-associated cues during GRPO training, alongside math questions that use a Python interpreter.
Change the cue, keep the question
Evaluate matched questions with and without the irrelevant cue. Extra search calls on math problems reveal shortcut behavior.
Reward necessary tool use
An LLM judge assesses each tool call. A penalty for unnecessary calls supplements the final-answer reward.
The cue changes the action
| Training | Test cue | Search calls | Accuracy |
|---|---|---|---|
| Cue-trained | Absent | 0.9% | 84.4% |
| Cue-trained | Present | 40.1% | 83.0% |
| + Tool-necessity reward | Absent | 0.2% | 88.8% |
| + Tool-necessity reward | Present | 0.2% | 92.8% |
Search is unnecessary for these math questions. Values are percentages from Table 1; they describe this controlled setting, not all agent tasks.
What this tells us
- Answer accuracy is not enough. Evaluating the actions an agent takes exposes failures that its final answer can hide.
- Shortcuts depend on what the agent learns. In these experiments, tool competence and the cue's semantic association with the tool help explain when shortcuts emerge.
- Tool decisions need feedback. Explicit necessity rewards reduce cue-driven search while preserving task performance.
Citation
@article{yang2026spurioustooluse,
title = {Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act},
author = {Yang, Yiwei and Zhang, Haoxiang and Wen, Bingbing and
Lu, Yao and Wu, Yuchen and Zhang, Lei and
McAuley, Julian and Lu, Pan and Howe, Bill},
journal = {arXiv preprint arXiv:2609.16268},
year = {2026},
url = {https://arxiv.org/abs/2609.16268}
}