22 Oct 2026 AI in test automation: where it saves time, and where you still need a tester

In short: AI in test automation saves real time on drafting test cases, writing first versions of scripts, making test data and reading failures. It makes a test suite worse when nobody checks what it writes. Then you get tests that always pass, more tests than anyone maintains, and flaky tests at scale. Let AI write, let people decide what to test and check the result.

Writing an automated test used to take an hour. With an AI assistant it takes a minute. That sounds like the end of slow testing, and for some tasks it really helps. But a test suite is only useful when people trust it, and trust doesn’t get cheaper when tests do. According to the Ranorex Software Quality Pulse Report, 53% of code is now written or assisted by AI. Only 17% of respondents say AI testing tools have had a significant impact. So where does AI help, and where does it create extra work?

Where does AI help in test automation?

AI helps most with work where a person can check the result quickly. Four tasks stand out. First, drafting test cases from a user story: AI reads the story and proposes cases, including edge cases a tired tester might skip. Second, writing the first version of a test script. Tools now do this out of the box; Playwright ships test agents that plan tests, generate the code and try to repair failing tests. Third, test data: realistic names, addresses and orders in the shape your system expects. Fourth, reading failures. AI can summarise a long log and point to the step where things went wrong, which saves time when a nightly run turns red.

In each of these, AI gives you a draft. A tester still reads it, throws out what doesn’t matter and adds what’s missing. We use AI where it helps our testers, and that is the pattern we see work: AI writes, a person decides.

Where does AI make your test suite worse?

AI makes a suite worse when its output goes in without review. Three problems show up again and again.

• Tests that always pass. A generated test can look complete and still check nothing. The assertion is missing, too loose, or checks a mock instead of the real system. These tests stay green whatever you break.
• More tests than anyone maintains. When a test costs a minute, teams add hundreds. Every one of them needs fixing when the application changes. Maintenance grows faster than the value.
• Flaky tests at scale. Generated scripts often wait for the wrong thing or depend on data that changes. One flaky test is annoying. Fifty make people ignore the whole run. We explained the causes in why automated tests fail when nothing changed.

Self-healing deserves a separate warning. A tool that repairs a failing test by itself saves time when a button moved. It can also hide a real change, such as a field that disappeared on purpose or by mistake. Then the test passes, and nobody notices that the behaviour changed. We’ll go deeper into self-healing in a later post.

How do you check that AI-written tests really test something?

You check AI-written tests the same way you check any code: you review them, and you try to make them fail. Review means a tester reads each test and asks one question: what would have to break for this test to fail? If the answer is “nothing”, the test goes. Then break the code on purpose. Change a calculation or a rule, run the tests, and see if one turns red. This is what mutation testing does automatically. In April 2026 Thoughtworks put mutation testing in the Trial ring of its Technology Radar, with this problem in mind. With AI-generated tests now common, it catches tests that stay green whatever changes.

Two more habits help. Start with business rules at the API level, where tests are fast and stable, before you generate long UI scripts. We wrote about that in API testing that guards your business logic. And keep the suite small on purpose. A compact regression set that runs every night and that people trust beats a large one that nobody reads.

What stays human work?

Deciding what to test and why stays human work. AI doesn’t know which part of your system would cost the most if it broke. It doesn’t know that finance closes the books on the last Friday of the month, or which customer always uses the old flow. Choosing the risks needs someone who knows the business. So does designing the few test cases that matter, and judging whether a strange result is a bug or a feature.

Exploratory testing and acceptance testing also stay with people. An exploratory tester follows their curiosity and finds the problems no script was written for. Acceptance testing with business users checks whether the software does what they need, not just what the story said. AI supports both, for example by suggesting ideas to try. It doesn’t replace them. For the wider picture of AI in testing, see our overview of AI in software testing.

What this means for your team

Use AI where it speeds up work a person can check. That means test case drafts, first scripts, test data and failure analysis. Review every generated test, try to make it fail, and keep the suite small. Decide upfront who owns the suite, because that is where the real cost sits. Writing tests got cheaper. Keeping them reliable, release after release, did not.

That last part is easy to underestimate. The suite runs every night, something breaks, and someone has to find out whether it’s the application, the data or the test. When the team is busy with the next release, that work slides, and the suite slowly stops being trusted. This is the part we pick up with Test Automation as a Service. We set up, run and maintain your regression tests alongside your team, and we use AI where it helps our testers. You keep control of what ships. One of our TaaS clients puts it like this: “We are able to release on demand thanks to the automated regression suite.” Read how it works on our Test Automation as a Service page.

Frequently asked questions

Will AI replace test automation engineers?

No. AI takes over parts of the writing, such as drafts of test cases and scripts. Deciding what to test, checking what AI produced and keeping the suite reliable still need an experienced engineer. If anything, that judgement matters more now that tests are cheap to make.

Which AI testing tools are worth trying?

Start with what fits your current stack. If you use Playwright, its built-in test agents are a low-risk way to try AI test generation. Measure the result: how many generated tests survive review, and how many fail when you break the code on purpose.

What is self-healing test automation?

Self-healing tools repair a failing test by themselves, for example when a button gets a new name or position. That saves time on small changes. Review every repair, because a fix can also hide a real change in how the application behaves.

Can AI write Playwright tests?

Yes. AI assistants and Playwright’s own test agents can generate working test code from a description or by exploring the application. Treat that code as a first draft: check the assertions, the waits and the test data before it joins your suite.