One customer, two partitions
Repeated entities cross the evaluation boundary and make generalization look stronger than it is.
Development case study / 2026
Can an agent fix the pipeline without breaking the experiment?
Faultline is a reproducible evaluation harness for agents repairing machine-learning system failures. It is built around the mistakes that make a model look better while making the underlying evidence worse.
05failure classes
11framework tests
02required outcomes per task
00headline model claims—yet
01 / THE CONTRACT
A benchmark should fail before the model does.
Each task begins as a small repository with a plausible ML defect. An agent receives the issue and public tests. The grader adds hidden checks only inside a temporary grading copy, then records what happened without confusing infrastructure trouble with model behavior.
01Prepareversioned public workspace
→
02Repairagent edits + complete trace
→
03Gradehidden deterministic checks
→
04Recorddigests, provenance, outcome
A checked-in task is valid only when the broken baseline fails for the intended reason and the reference repair passes the combined public and hidden suite. Integration tests enforce both sides of that contract for every task.
02 / CURRENT TASKS
Five ways a clean score can lie.
The retry changes the ranking
Micro-averaging lets frequently retried tasks outweigh every other task in the benchmark.
A seed with side effects
Initialization mutates the training data and contaminates randomness elsewhere in the process.
A label from next month
A historical split trains on outcomes that were not observable at the evaluation cutoff.
Inference quietly retrains preprocessing
Online standardization uses the request batch instead of the statistics learned during training.
03 / THE AUDIT
The first broken task was mine.
The entity-split prompt originally asked for a test set that was “reasonably close” to the requested size. Its hidden grader enforced a numerical tolerance the agent could not infer. That made the evaluation underspecified.
BEFORE
“Reasonably close” in the issue; an unstated tolerance in the grader.
AFTER
Closest attainable complete-entity partition, with the tie-break stated and tested directly.
I kept the correction in the evaluation card because this is the work: not merely writing tests, but asking whether the tests have earned the right to decide.
04 / WHAT COUNTS
No leaderboard before the benchmark is ready.
Measured now
- Task and workspace content digests
- Public and hidden deterministic outcomes
- Human and assistant provenance
- Timeouts and infrastructure failures
- Baseline-fail / reference-pass validity
Publication gate
- At least 20 independently audited tasks
- A protected holdout or procedural variants
- Repeated trials and uncertainty intervals
- Cost, latency, and invalid-run reporting
- Blinded human review of sampled trajectories
AI COLLABORATION
I use coding agents while building Faultline and record that assistance in run manifests. I remain responsible for the research question, task validity, accepted changes, independent reproduction, and every published claim.
05 / NEXT EXPERIMENT
Run agents, study the misses, then decide what deserves training.
The next release adds complete trajectory capture, repeated trials, cost and latency, and a larger audited task set. Only after that evaluation is stable will failure traces become candidates for a held-out post-training study.