A generated test that calls a function and checks nothing still turns a red coverage bar green. That’s the whole problem, in one sentence.
Coverage percentage measures which lines executed during a test run. It says nothing about whether anything meaningful got checked once those lines ran. Point an AI coding assistant at an uncovered function, ask it to “add tests,” and it will happily write something that executes every branch and asserts almost nothing worth trusting. Ask it twice and it’ll do it twice, in under five minutes both times. Why does that keep happening? Because the model is optimizing for the number you gave it, not for the thing the number was supposed to represent.
What a coverage-passing, assertion-free test actually looks like
Here’s a Python example that would raise coverage on calculate_discount from 40% to 100% without verifying a single thing about the discount logic:
def test_calculate_discount():
result = calculate_discount(100, 0.2)That’s it. No assert anywhere. The function runs, the line executes, the coverage tool marks it green. If calculate_discount returned None, or raised a KeyError that got silently swallowed somewhere upstream, this test would never know and would never fail.
The slightly more sophisticated version wraps the call in a try/except and asserts only that nothing blew up:
def test_process_order_does_not_raise():
try:
process_order(sample_order)
except Exception:
pytest.fail("should not raise")Same story in JavaScript. Vitest will happily report a passing, fully covered test that checks the shape of a value without checking its contents:
it("calculates the discount", () => {
const result = calculateDiscount(100, 0.2);
expect(result).toBeDefined();
});toBeDefined() passes for 0. It passes for -4000. It passes for an empty object. In my own reviews I’ve watched a handful of these slip into a single sprint’s PRs the moment coverage became a merge gate, and each one looked, at a glance, like a real test.
Why skimming the diff doesn’t catch this
Here’s the uncomfortable part: this pattern is genuinely hard to spot in code review. The test file looks completely normal. It has a sensible name, a real function call, plausible fixture data, sometimes even a comment explaining what it’s supposedly checking. A reviewer scanning thirty added lines across a PR that touches four other files is not going to stop on line 14 and ask whether this specific assertion would fail if the underlying logic were wrong.
Would you catch it, every time, on a Friday afternoon with six other PRs waiting? Probably not. That’s not a knock on any one reviewer. It’s a structural weakness in judging tests by reading them instead of by trying to break them.
What coverage tools were actually built to measure
Martin Fowler’s classic essay on test coverage, written in 2012 and still the clearest word on the subject, makes a point that gets lost constantly: coverage is a tool for finding code nobody tested at all, not a score for how good the existing tests are. A codebase can sit at 95% coverage and still ship a bug in the one line a test happened to execute without checking. Coverage answers “was this code run.” It has no opinion on “was this code verified,” and it was never built to.
That gap used to be mostly academic, the kind of caveat you’d mention once in a code review and move past. It stopped being academic the moment coverage became a number a model could hit on demand, faster than any human ever bothered to game it by hand. Writing an assertion-free test used to take almost as much effort as writing a real one. Now it’s nearly free.
Mutation testing: asking the only question that matters
So what actually tells you whether a test is worth anything? Not the coverage percentage. Mutation testing does, by asking a completely different question: if this code were wrong, would your tests notice?
The tool takes your source, introduces a small deliberate bug (flips a > to >=, swaps a + for a -, deletes a line entirely), and reruns your suite against that mutated version. If the suite still passes, the mutant “survived”, and that survival is the tool telling you, in plain terms, that nothing you wrote would have caught this exact bug in production either.
Stryker Mutator does this for JavaScript and TypeScript, and its own documentation describes precisely the failure mode this article is about: a suite sitting at 100% line coverage can still let most introduced mutants survive, because line coverage never checked what the assertions actually verified. Python has mut.py and cosmic-ray for the same job. Java teams reach for PIT, which has been doing rigorous mutation analysis on the JVM since long before anyone needed a phrase like “AI-generated tests.”
None of this is exotic tooling locked behind a research paper. Wiring one of these into CI is a weekend of work, not a quarter-long initiative, and it is the fastest way I know to make an assertion-free test visible instead of invisible.
One honest caveat before you turn this on for a mature codebase: mutation testing is slow. It reruns your whole suite once per mutant, and a few thousand tests against a few thousand mutable lines can take hours, not minutes. Most teams I’ve seen run this well don’t run it on every single PR. They run it nightly against the full codebase, or scope it to just the files a given PR touched, and gate merges on a smaller, faster mutation score for the diff alone.
The second signal: does the test even pass reliably
Mutation score answers one question. It leaves a related one wide open: is this test deterministic? A test that passes 19 times out of 20 because it depends on wall-clock timing, unseeded randomness, or a race between two async calls isn’t trustworthy either, whatever its coverage or mutation score claims. How do you know a green run wasn’t luck? You don’t, unless you run it more than once.
The fix is unglamorous. Run new or changed tests N times, 20 is a reasonable default, in a loop before merging, and flag anything that doesn’t produce the same result every single time. Vitest supports a --repeat flag built for exactly this. Pytest has pytest-repeat. Neither takes more than an afternoon to wire into a CI job scoped to files touched in the current pull request, which keeps the extra runtime bounded instead of tripling your whole pipeline.
What actually changes on a team that adopts this
Here’s my genuinely disagreeable opinion: a bare coverage percentage in a pull request template is worse than no number at all, because it invites exactly the gaming this article describes while handing everyone a false sense that the suite improved. A mutation score, even a mediocre one, tells the truth in a way a coverage badge never does.
None of this replaces human judgment about what deserves a test in the first place. It replaces the illusion that a percentage on its own proves anything. If you’re already working out which of your twelve CI gates should catch this first, the mutation-score check belongs right after your linter, not buried behind security scanning where nobody looks at it until a postmortem. And if your team has already drifted into quietly suppressing failing checks instead of fixing them, that same habit will swallow a mutation-testing gate exactly the way it swallows a lint rule.
The failure mode isn’t a mystery anymore. What’s left is whether anyone bothers to measure the thing that actually matters instead of the thing that happens to be easy to hit.