AI can generate tests extremely fast. Tools like GitHub Copilot, Claude, or similar assistants can produce hundreds of test cases, automation frameworks, and utilities in minutes. However, most teams quickly discover that faster test generation does not automatically translate into higher software quality.
The core issue is simple: AI scales artefact production much faster than teams scale validation maturity.
We spoke with Ostap Elyashevskyy, our Test Automation Competence Manager, to find out where AI really helps in quality engineering and where it does not live up to the hype.
Do more tests actually mean better quality?
Having more tests does not always lead to better coverage or improved risk detection. When AI operates with incomplete context, such as missing requirements, architecture decisions, or domain knowledge, its output tends to be generic and often redundant. In real projects, only some AI-generated tests added real value, while the rest duplicated scenarios or validated trivial flows.
Where does AI create the most value in test automation?
From a test automation architecture perspective, the real opportunity for AI lies not in test generation but in system analysis.
AI-powered audits allow large automation repositories to be analysed quickly. Instead of manually reviewing hundreds of files, an AI-assisted audit pipeline can inspect test code, configuration files, CI/CD pipelines, and supporting utilities, and then generate structured reports with quantified quality metrics.
A typical audit evaluates multiple dimensions of the test ecosystem, including:
- reliability and flakiness patterns
- test design and readability
- framework architecture consistency
- coverage maturity
- test data strategy
- CI/CD integration
- reporting, and observability
How does the audit pipeline work?
The audit pipeline typically follows a structured process.
First, repository discovery identifies frameworks, tools, and project structure. Next, pattern detection scans for anti-patterns such as excessive sleeps, implicit waits, fragile locators, or shared global state. Then, the structural analysis evaluates test architecture patterns, Page Object implementation, and test data handling strategies. Finally, a scoring model aggregates findings into a numerical assessment across multiple quality areas.
This approach combines prompt orchestration, schema-driven outputs, and scorecard-based evaluation models to ensure consistent results.
Why move from monolithic prompts to skill-based pipelines?
An important architectural decision is moving from monolithic prompts to skill-based pipelines. Instead of asking an LLM to perform an entire audit in a single prompt, the system executes multiple focused skills such as repository discovery, flakiness detection, coverage analysis, CI/CD inspection, and final report aggregation.
This significantly improves reliability, reduces hallucination risk, and makes the audit process easier to maintain.
| Aspect | Monolithic Prompt | Skills-Based Prompt |
|---|---|---|
| Reasoning Guidance | Model must infer how to analyse | Skills explicitly guide analysis |
| Hallucination Risk and Variability | Higher | Lower due to evidence and rules |
| Maintainability | Hard to update | Easy to update individual skills |
| Reusability | Low | High skills reused across agents |
| Large Repo Analysis | Often inefficient | Efficient via sampling and staged analysis |
| Debugging | Hard to identify issues | Easy to isolate problematic skill |
Another practical optimisation is to pre-filter source code with tools like grep before sending snippets to the LLM. This reduces context size and improves analysis efficiency, often decreasing token usage by an order of magnitude.
Should auditing be fully automated or human-led?
In practice, the most effective approach is AI-assisted auditing rather than fully automated analysis.
Manual audits by senior automation architects can take several days. Pure AI analysis is much faster, often finishing in minutes, but it usually misses important context. Using a hybrid approach, where AI handles large-scale analysis, and experts review the results, usually cuts audit time to a few hours while maintaining high reliability.
- All AI-generated artefacts should be validated by an expert; do not use it "as-is"
- Use AI where possible/experiment with prompts, approaches → the only way to learn and have a boost
- AI weaknesses: hallucinations are possible, inconsistent results between runs
| Code audit with report | Expert | AI-only | AI-assisted (AI + Expert) |
|---|---|---|---|
| Speed | Days (5?) | Minutes (15–30) | Hours (1–4) |
| Quality | High | Low-Medium | High |
| Variability | Low | Medium-High | Low |
| Recommended approach? | Yes | No | Yes |
How are audit results reported?
The audit results are generated as a structured report that can be exported in multiple formats depending on the audience and use case. The core output is a JSON report that serves as the source for generating other formats, including Excel (XLSX) with visual charts and score breakdowns, Word (DOCX) reports for detailed documentation, HTML dashboards for quick sharing, and presentation slides for management reviews. This flexibility allows the same audit data to support both technical deep dives for engineers and high-level summaries for stakeholders.
Related insights
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.
