A control that has only ever passed is not a control

The assertion that could never fail

An agent working on a three-day autonomous build wrote a check specifically to catch a real failure mode: database tables missing the security policies meant to protect them. The check ran, returned zero problems, and stayed green for days. The database underneath it was completely broken, missing table-level grants on every one of eighty tables, so that not one of its hundred and forty one row-level security policies had ever actually been evaluated. The first version of the check had filtered on a column that held the same value on every row, so it returned zero rows regardless of what the database actually looked like. It was not a weak check. It was an assertion that could never fail, which is worse than no assertion at all, because it converts an unknown into a false certainty and puts a green tick in front of it. The fuller story, including how the hole was found, is here: the green light that meant nothing.

The discipline: plant, fail, revert, pass

The fix for this is not a cleverer assertion. It is a discipline applied to every check that matters: plant a defect on purpose, confirm the check actually fails against it, revert the defect, confirm the check goes back to green. A check that has only ever passed has not been proven to do anything. It might be watching the wrong column, the wrong table, or nothing at all, and green tells you exactly the same thing either way. Twenty four static checks in that build were each put through this before they were trusted, and the practice caught problems that reading the checks themselves never would have surfaced.

What counts as a planted defect depends on what the check claims to catch. For a policy check, temporarily drop the grant it depends on and confirm the check reports the gap. For a cap or a limit, write a value past it directly and confirm the check objects. For a static analysis rule, write the exact pattern it is meant to flag and confirm it gets flagged. The defect has to be the real failure the check exists for, not a nearby approximation, because a check that catches a different, easier problem will still look proven while the actual gap stays open.

The glob that matched nothing

A file-matching utility, written to stand in for a runtime API that was not available in the build environment, contained a pattern-translation bug that meant the glob it produced could never resolve to an actual file path. Three separate verification scripts built on top of that utility were scanning an empty set on every run and passing cleanly, because an empty set has no failures in it by definition. Nothing about the passing output distinguished a genuinely clean codebase from a check that was silently checking nothing. It was caught only because someone applied the same discipline as above: the negative test, written specifically to confirm the check could fail, failed to fail.

Why this bites harder with agent-written code

A human writing a shortcut or cutting a corner under deadline pressure tends to leave a mark. The variable name is a little off, the comment is defensive, the logic reads as rushed. Reviewers have spent years learning to smell that. Agent-written code does not carry the same tells. It is fluent, idiomatic and confidently structured whether it is correct or not, because fluency is what a language model optimises for regardless of correctness. A privilege escalation path sat inside a function that read, on the page, like completely ordinary approval logic: it took the acting user as a parameter, and simply never checked that the caller actually was that user, while running with definer privileges that bypass row-level security by design. Nothing about the diff looked wrong. A business rule capping how many recipients could receive a given record was implemented correctly inside the function that allocated them and enforced nowhere else, so the underlying column stayed directly writable by anything that could reach it. Again, nothing about the diff looked wrong.

Both of those were found by running the software and trying to break it, not by reading it. That is the actual argument for negative testing on AI-generated code specifically: the review step most teams rely on to catch a bad shortcut is calibrated to catch human tells, and this code does not have them.

The question to put to any team

Skip the audit of every check in the codebase and ask one question instead: can you show me this check failing? It takes minutes to answer for any single check, and a team that has actually done this work will have the answer ready without needing to go and check first. Pick any assertion someone points to as proof the system is safe, plant the exact defect it claims to catch, and run it. If it fails, the control is real. If it passes, you have found the same failure mode as the story at the top of this page, a check that has never once been tested against the thing it exists to catch, and no amount of it staying green tells you anything at all.

The other side of this same build produced a working model for governing what an agent is allowed to do in the first place, not just how its output gets checked afterwards, covered here: why an instruction file is not the same thing as a control.