AI-generated unit tests: where they help, and when they lie to you
Jul 20, 2026 · 9 min read · By Daniel Reyes, Engineering
AI-generated unit tests are genuinely useful and genuinely easy to misread. A model can produce a test file for an untested module in seconds, and coverage jumps. The problem is that coverage measures which lines ran, not whether anything meaningful was checked, and a generated suite is unusually good at producing tests that execute code without asserting anything that would ever fail.
Used well, generated tests are the fastest way to build a safety net around code that has none. Used carelessly, they are a number that makes everyone comfortable while catching nothing.
What generated tests are actually good at
- Covering the boring paths. Null inputs, empty collections, boundary values, error branches. Real defects hide here and humans skip them because they are tedious.
- Characterizing untested legacy code. Capturing what the code does today so you can change it safely. This is the highest-value case.
- Filling gaps on a change you just made, where the intent is fresh and the diff is small.
- Volume. A model does not get bored on the fortieth case, which is exactly where human test writing degrades.
The three ways a generated test lies
1. The tautology
The test calls the function, then asserts that the result equals what the function returned. It passes forever, including after you break the code. These appear when the model does not have enough context to know what the correct output should be, so it encodes the observed output as the expectation without any independent notion of correctness.
The tell: an assertion that could not fail unless the function throws. If you cannot describe what bug the test would catch, it is not a test.
2. The frozen bug
The code has a defect. The model reads the code, infers that the current behavior is intended, and writes a test asserting the buggy output. Now the bug has a test defending it, and the next engineer who fixes it sees a red suite and reverts.
This is the most dangerous failure because it looks like diligence. It is also why generating tests from the code and generating tests from the requirement are different activities. The first characterizes; only the second validates. Both are useful, but do not confuse one for the other.
3. The mock that tests the mock
Everything is stubbed, so the test verifies that the mocks were called in the order the test set up. It passes whether or not the real integration works. Heavy mocking is often a sign the model could not figure out how to exercise the real path.
How to review a generated test suite
The review question is not "does it pass." It is "would it fail if the code were wrong."
| Check | What you are looking for |
|---|---|
| Mutate the code | Break the function deliberately, flip a comparison or change a return value, and rerun. Any test that still passes is not testing that behavior. This is the single most effective check. |
| Read the assertions, not the count | Does each one express an expectation you would defend in review, or does it restate the implementation? |
| Check what is mocked | If the interesting logic is stubbed, the test is measuring the stub. |
| Look for the failing cases | A suite where nothing tests error handling has covered the happy path and called it done. |
| Ask about the requirement | For any assertion encoding business logic, confirm it matches the spec rather than the current code. |
The mutation check is worth institutionalizing. It takes two minutes and it is the difference between a suite that protects you and a number on a dashboard.
What to do with the tests you reject
Do not silently delete them. A generated test you throw away is usually telling you something about the code. A tautology often appears because the function has no clear contract, so there was nothing to assert against. Heavy mocking usually means the unit has too many dependencies to exercise honestly. A model that could not produce a meaningful test for a function is a reasonable signal that a human would struggle to write one too, and that the function is doing too much.
Treat the rejects as a list of refactoring candidates rather than as noise. On more than one occasion the useful output of a test generation pass has been the map of which modules resisted testing at all.
Coverage is a bad target and a decent diagnostic
Once coverage becomes a goal, generated tests will hit it without improving anything, because the cheapest path to a high number is tests that execute code without asserting much. Coverage is useful in one direction only: low coverage reliably tells you something is untested. High coverage tells you nothing about quality.
If you want a number that means something, track escaped defects, or how often a change breaks a test that should have caught it. Teams that watch data pipelines already know this pattern, where the useful signal is not how many checks are configured but whether a schema change gets caught before it reaches a dashboard. Same principle: the value is in what gets caught, not what gets counted.
Where Agentcode fits
Agentcode writes tests as part of doing a task rather than as a separate coverage exercise, which sidesteps the worst version of this problem. When it changes code, it adds the tests covering that change and runs your entire existing suite. If anything fails, no pull request is opened.
For bug fixes it goes further: the regression test has to fail on the original code and pass on the fixed code. That is a built-in mutation check, and it means a test that would pass either way does not count as proof. See how the agent runs your real test suite and how every fix ships with a regression test as evidence.
The short version
Generate tests to characterize code you are about to change, and review them as a specification rather than as output. Delete the tautologies. Verify the assertions against the requirement, not the implementation. Break the code on purpose and confirm the suite notices. A generated suite that survives that treatment is a real safety net, and it took an afternoon instead of a sprint.
If you are weighing how much of this work to hand over, the practical framing is in whether your team needs an AI coding agent, and the review side is covered in how AI code review works and where it falls short.