Why does an AI agent reading only your codebase produce a false signal?
Picture an AI agent that can crawl your repository, map out the components and endpoints, write a stack of Playwright tests, run them, and even patch the ones that fail. Watching a green pipeline appear feels like magic. That’s exactly the moment to be suspicious.
Here's the structure underneath it:
implementation > generated expectation > test of the same implementation
If your code has the wrong route, is missing a required field, or has a limit set incorrectly, an agent that only reads the codebase will copy that mistake right into the test’s expectation. The test passes, not because the product is correct, but because both sides of the check came from the same flawed source.
That’s basically the representation of the test oracle problem, which was formalised by Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo in a 2015 survey. Their main point still stands even with the usage of AI.
A test only means something when the expected result comes from somewhere independent of the code being tested:
- Expected result comes from the approved requirement.
- Actual result comes from running the application.
- If they disagree, that’s a real, useful failure.
That’s the whole idea. Everything below is just working out the consequences.
As for the repository codebase, it’s genuinely useful for automation. Use it to learn the folder structure, existing fixtures, how authentication works, page-object conventions, API clients, selectors, and how to actually run the tests. At the same time, it is the wrong source for what the product is supposed to do: which fields are mandatory, the correct upload limit, where a button should navigate, who’s allowed to do what, what an error message should say, whether something missing is actually a bug. Requirements define correctness. Code implements it. Tests are supposed to compare the two. Generate both sides from the same source, and there’s nothing left to compare.
What patterns actually show up in these "always-green" suites?
“Always green” doesn’t mean the tests can never fail; they’ll still break on selector changes, flaky timing, and refactors. The real issue is a bias: these tests are built to confirm whatever the implementation currently does, defects included, and they tend to survive exactly the changes that should have broken them.
A few patterns show up constantly.
The generator copies constants straight out of the code. If production defines a 12 MB upload limit, the generated test checks for 12 MB, even if the requirement says 10.
export const MAX_ATTACHMENT_SIZE_MB = 12; // wrong, requirement says 10
test('shows the attachment size limit', async ({ page }) => {
await expect(page.getByText(`Max. file size: ${MAX_ATTACHMENT_SIZE_MB} MB`))
.toBeVisible();
});
Both the app and the test agree with each other. Neither agrees with the requirement.
It can only test what’s actually there.
Say a contact form is supposed to require six fields, but the developer forgot "Phone number." An agent scanning the component's markup sees five input elements and writes five checks. Coverage looks great. The missing field is invisible, because you can't generate a test for behaviour that was never implemented.
Self-referencing assertions turn into tautologies.
This one shows up more than people expect:
const currentTitles = await cards.getByRole('heading').allTextContents();
await expect(cards).toHaveCount(currentTitles.length);
The expected count and the actual count are read from the same place. Zero cards on the page? Still passes.
Snapshots freeze whatever was there when the screenshot was taken.
A visual baseline answers “did the page change,” not “was the page correct in the first place.” Approve a snapshot after a bug ships, and the bug is now the permanent baseline.
An unsupervised healer can quietly erase real regressions.
Say the requirement locks the “Contact us” link to /contact-us/, but someone changes it to /get-in-touch/. A requirement-first test correctly turns red. A healer that’s allowed to “fix” failing tests by matching whatever the app currently does will just rewrite the assertion to /get-in-touch/ and report success. That’s not healing; that’s deleting the evidence of a regression.
None of this is really an AI hallucination problem. It’s an architecture problem: nobody gave the generator an independent source of truth.
How do you stop AI from writing circular tests?
A test-generation setup that actually works keeps three inputs separate and never lets one quietly overwrite another:
| Input | What it's for | What it must never redefine |
|---|---|---|
| Approved requirements | Expected behaviour, the actual oracle | Implementation details |
| Codebase | Structure, fixtures, technical feasibility | Business expectations |
| Running application | Locators, real runtime behaviour | Approved expected behavior |
In practice: expected results come from the requirement. Test steps and structure come from requirements plus code. Pass/fail is a comparison between the expected requirement and the observed runtime, never code against code. This also gives you a clean decision table for triage:
- Requirement and code agree, and the app behaves accordingly → pass.
- Requirement says X, code does Y, app behaves like Y → fail, and it’s probably a product defect, not a test bug.
- Requirement is ambiguous → stop and ask; don’t let the agent guess and call it approved.
Feed it requirements, not a Jira ticket and “go generate tests”
Dumping an entire repo and a vague ticket on an agent with “generate all relevant tests” is how you get the mess above. Start by turning requirements into a small, structured artefact: include an ID, the approved statement, its source, and the key values that matter, such as a size limit or a list of required fields. Each generated test case should include that requirement ID. This way, if a requirement changes later, you can trace everything downstream and mark it as stale, rather than letting things quietly break.
Repo layout matters too: keep requirement artefacts and test-design files clearly separate from src/, so it’s obvious at a glance that a test’s oracle didn’t come from the implementation.
6 key guardrails for the AI itself
Whatever prompt or agent setup you’re using, spell out explicitly:
- Requirements define expected behaviour; the repo defines conventions and helpers; runtime confirms actual behaviour, and none of these substitute for another.
- Keep every requirement ID attached to its test.
- If the running app contradicts a requirement, keep the requirement-based assertion and report the failure; don’t rewrite it to match.
- Never derive an expected business value from a production constant.
- Never build an expectation by reading the same UI value you’re about to assert against.
- If information is missing, stop and ask instead of guessing.
Playwright’s own planner/generator/healer agents are genuinely useful for structure, locators, and mechanical repairs, but just don’t solve the oracle problem on their own. A healer should first classify a failure (test bug, product defect, requirement changed, bad test data, environment issue) and only auto-fix the first category and a narrow slice of the third. URLs, limits, roles, and business messages should never get silently rewritten by an automated healer.
What to track instead of vanity numbers?
Test count, pass rate, and code coverage feel good but don’t tell you whether the suite catches real defects. Track requirement coverage instead — the percentage of assertions with an independent oracle, mutation score, defect-detection rate, and how many “healed” tests actually turned out to be masked product regressions. The single most useful question you can ask a suite: if I deliberately break a requirement, does the right test go red?
The bottom line
A generator that only looks at the codebase really just checks if the product matches its current implementation. That question is always going to give a comforting answer. What really matters, though, is whether the implementation meets what the business agreed to.
Technical polish, like clean TypeScript, strong locators, fast parallel runs, and clear traces, does add value, but it can’t replace an independent source of truth. Requirements define what’s correct. Code tells you how to check it. The running app shows the real result. The test compares them. And a person still needs to resolve any confusion and approve changes that affect what a test means, not just how it’s written.
Related Insights
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.
