Field note / Agent evaluation
Tool-call correctness has four surfaces.
A single pass/fail score hides where an agent failed and where an evaluator was blind.
When an agent uses tools, correctness is distributed across the decision to act, the call it constructs, the system result, and the explanation it gives the user. Treating those as one surface makes failures harder to detect and diagnose.
1. Tool choice and cardinality
Did the agent select an allowed tool? Did it call every required tool and avoid undeclared ones? Did a side effect happen exactly once? This surface catches missing calls, extra calls, invalid ordering, and repeats. It matters most when tools mutate state.
2. Arguments
A correct tool name can carry a dangerous payload. Evaluate required fields, values, types, and domain constraints. For a refund, amount and order identity deserve explicit assertions. For a search tool, looser semantic comparison may be appropriate. Match strictness to consequence.
3. Execution outcome
A well-formed call can still fail. The evaluator must see whether the tool succeeded, partially completed, timed out, or returned a business error. Otherwise it can reward an agent for attempting an action that never happened.
4. Final claim
The response should agree with observable state. “Your refund was issued” is incorrect after a failed tool result even when the tool choice and arguments were perfect. Conversely, a correct action paired with a confusing answer is still a product failure, but it is a different one.
Build the evaluator from the action contract
Start with the system’s authority boundary. List allowed tools, required calls, non-idempotent operations, high-risk arguments, expected result states, and claims the response may make for each state. Those become assertions before they become a score.
Then pressure-test those assertions. The 263-run evaluator comparison shows what can disappear when only the response is visible. Mutation testing turns each surface into a falsifiable question: if this known fault is planted, does the verdict change?
Use scores for summaries, survivors for work
An aggregate score helps compare versions, but the actionable unit is the surviving fault. “Argument corruption survived for issue_refund.amount” tells the team which contract is missing. Preserve that detail in CI output and regression history.