Field note / Agent evaluation
A response-only eval missed 176 of 263 broken agent runs.
The final answer is only one surface of an agent run. A pinned mutation set made the cost of ignoring the others measurable.
A fluent final answer can hide a missing call, the wrong argument, a repeated side effect, or a tool error. I wanted to know how much an evaluator loses when it judges the answer but not the execution that produced it.
The comparison
I used Mendmark to generate a pinned set of 263 broken runs from passing agent cases. Every run contains a controlled fault, and every evaluator receives the same set. The comparison is deliberately narrow: it measures sensitivity to these mutations, not general agent safety.
The response-only evaluator caught 87 mutations and missed 176. An exact trace evaluator caught more because it could inspect tool names, arguments, order, and call counts. A trace-plus-outcome evaluator caught all 263 in this pinned set because it also compared tool results and the final response with the expected outcome.
Why the final answer looked convincing
Language models are good at producing coherent explanations. That creates a dangerous asymmetry: the response can remain plausible after the underlying action has changed. A refund agent may say the refund succeeded when the tool returned an error. It may report the expected amount after sending a different amount. It may call the refund tool twice and mention only one transaction.
A response-only judge cannot reliably infer hidden execution state. Asking it to do so turns an observable correctness question into a guess.
What to evaluate instead
Start with the contract of the task. Which tool had to run? Which arguments were required? How many side effects were allowed? What result constitutes success? What must the final response say—or avoid saying—given that result?
Then test the evaluator itself. Remove a required call. Corrupt a high-risk argument. Repeat a non-idempotent action. Replace a successful tool result with an error while preserving the final answer. Each known failure should change the verdict from pass to fail.
Reproduce before generalizing
The useful part of this result is not the headline percentage. It is the survivor list. A survivor identifies an obligation the evaluator did not enforce. Teams can reproduce the golden set, inspect each operator, and replace the sample policies with contracts tied to their own tools.
That is why the Mendmark case study publishes the evaluator breakdown and limitations, and why the repository keeps the mutation set pinned. Claims about eval quality should be inspectable.