Your agent eval passed. Would it catch a broken tool call?

Imagine a customer asks an agent to refund order 104. A correct run looks up the order, calls the refund tool once with the right amount, and tells the customer what happened. The eval passes. The team ships.

Now change the trace in four small ways:

1. Remove the refund call.
2. Change the refund from $29.99 to $299.99.
3. Call the refund tool twice.
4. Return an error from the tool while the final answer still claims success.

Run the same eval after each change. If it stays green, the agent is no longer the immediate problem. The test cannot recognize a failure the team already knows about.

Test the scoreboard

Software teams have used mutation testing for years. A mutation tool changes working code in a controlled way, then checks whether the test suite notices. A test that catches the change kills the mutation. A test that misses it leaves a survivor.

Agent evals need the same pressure. Their inputs are different. The dangerous change may live in a tool argument, a repeated side effect, a hidden tool error, or an answer that no longer matches the action taken.

What Mendmark does

I built Mendmark to make this check repeatable. It begins with a passing agent case and the tools that case is allowed to use. It creates controlled mutations, reruns the team’s existing evaluators, and reports what they missed.

The current operators can:

• Remove a required tool call.
• Add an undeclared tool.
• Change, remove, or corrupt a tool argument.
• Corrupt a tool result.
• Repeat a side effect.
• Reorder the trace.
• Hide a tool failure behind a success message.
• Remove or damage the final response.

The report names each survivor. It also tracks coverage by tool, notices when a tool contract changes, and can fail CI when a serious fault survives or an old detection disappears.

What the score means

A mutation kill rate describes the eval suite against the faults that were planted. It does not prove the agent is safe, correct, or ready for production. Unknown failures still exist. Product-specific risks still need product-specific tests.

The useful output is often the survivor itself. “Our refund eval ignores the amount” gives an engineer something concrete to fix. “The agent scored 0.87” does not.

Why this matters now

Agents can act through tools. A weak answer is annoying. A repeated refund, deleted record, or false claim of success can change the state of a real system. Every new tool gives the agent another way to fail and gives the eval suite another obligation.

Teams already test their agents. The next question is whether those tests deserve the authority they have been given.