Knowing when to ask
Real requests to coding agents are often missing a key detail. A good agent should notice and ask, but only when the question is worth the user's time. This project builds a benchmark from real agent sessions to measure whether coding agents ask the right clarification questions, and how much it costs when they don't.
Preliminary results on the current 171-task benchmark. Numbers may change as the work continues.
From real sessions to tasks
How the benchmark works
Find missed questions
Identify sessions where the agent went ahead on an underspecified request and the user later had to correct it.
Rebuild the task
Restore the repository state for each case and pair it with a sandbox and an executable verifier.
Simulate the user
A simulated user returns the original user's correction only when the agent asks about the missing specification.
Price each question
Score each question by its cost of not asking: tokens spent on tool calls and line edits along the wrong implementation path, plus the cost of undoing it.
Early observations
- Agents rarely ask. Fable 5.1 asked for clarification in only 18% of runs, even though every task is missing information the user cares about.
- When they ask, they don't ask well. Its decisions to ask were only weakly calibrated to question value: 60% of its questions were low-value, against an 80% base rate.
- Not asking is costly. The final-state pass rate was 23%, and wrong guesses lead to wasted tool calls and edits that have to be undone.
Status
This project is ongoing, and a paper is in preparation. If you are interested in the benchmark or would like to collaborate, please get in touch.