87 caught. 176 missed.
The final answer exposed some failures, but 162 critical tool-behavior faults still passed.
Open source / PyPI 0.4.2 / 2026
Your agent passed the test. Would the test notice if it broke?
Mendmark plants controlled faults in passing agent traces, then runs the same evals again. A missed tool call, wrong argument, repeated side effect, or hidden error becomes a concrete test gap instead of a surprise in production.
24reviewable golden cases
263pinned mutations
10operating domains
100%complete-profile kill rate
01 / THE TEST OF THE TEST
Plant a fault. See if the eval catches it.
A passing case is the starting point. Mendmark changes one part of its output or tool trace, then reruns the team’s existing evaluators. The mutation is killed when an eval changes from pass to fail. If every eval still passes, Mendmark names the blind spot.
01Loadpassing cases + tool contracts
→
02Mutateone controlled fault at a time
→
03Rerunthe same evaluator suite
→
04Gateblock survivors and regressions
The resulting kill rate measures the eval suite, not the agent. A surviving critical fault is a specific evaluator problem the team can reproduce and fix.
02 / PUBLIC GOLDEN SET
Three evaluators. The same 263 broken runs.
The versioned golden set contains 24 passing cases, 13 tool contracts, and 39 tool calls across search, commerce, payments, CRM, communications, databases, scheduling, weather, support, and multi-step workflows. Every case and expected trace is public.
215 caught. 48 missed.
The ordered calls were correct, but removed or replaced final responses escaped.
263 caught. 0 missed.
Exact checks of the ordered tool trace and final outcome detected every planted fault in the set.
SCOPE
This result measures evaluator sensitivity to these controlled faults. It does not establish overall agent accuracy or safety. The benchmark is deterministic, runs offline without model calls, and is pinned so an unreviewed change to its cases, mutations, or expected results fails CI.
03 / CONTROLLED FAILURES
What Mendmark breaks on purpose.
A required call disappears
Mendmark removes each tool call and adds an undeclared tool to see whether the trace still passes.
The right tool gets the wrong input
One argument changes at a time, exposing evals that check the tool name but ignore its payload.
The payment happens twice
Calls marked as side effecting are duplicated to test whether the suite notices unsafe retries.
The tool fails. The agent says it worked.
A successful result becomes an error while the final response still claims completion.
The answer vanishes or changes
The response is removed or replaced to confirm that outcome checks are doing real work.
04 / TOOL ROLLOUTS
A new tool should change the release conversation.
Mendmark hashes every declared tool contract. When a tool appears or its schema changes, the report shows whether any case exercises it and whether those evals catch faults in its calls. Basic schema checks also find missing required arguments and wrong value types without recording the values themselves.
BEFORE
A pull request adds a refund tool. The aggregate agent score stays green.
AFTER
The report marks the new side effect, shows its mutation coverage, and blocks any critical survivor.
A committed baseline remembers which faults the suite caught before. If a changed metric lets one of those faults pass, CI reports the exact regression.
05 / WHAT SHIPS TODAY
A released local engine with a small data footprint.
Open-source engine
- DeepEval suite adapter
- Framework-neutral JSON adapter
- Nine built-in mutation operators
- Per-tool mutation coverage
- Tool contract and schema tracking
- Regression-aware CI gates
Report boundary
- JSON, JUnit, and SARIF output
- Custom mutation plugins
- Source, policy, and CI provenance
- Signed report verification with Cosign
- PyPI Trusted Publishing provenance
LIMIT
A passing mutation audit does not prove an agent is safe. It shows that the configured eval suite caught the faults Mendmark tested. Teams still need product-specific cases and mutations for their highest-risk behavior.
06 / TRY IT
Run a complete audit without an API key.
The repository includes both a small refund-agent demonstration and the public golden set with 263 pinned mutations. Clone the project and reproduce the results without an API key. Teams with an existing suite can use the DeepEval adapter or connect any local evaluator through the versioned JSON protocol.