06 Oct 2026 Flaky tests: why your automated tests fail when nothing changed

In short: Flaky tests are automated tests that pass one run and fail the next, while nothing in the code has changed. Timing, test order and outside services cause many of them. In business systems the biggest cause is often test data that other people and systems keep changing. Fix the data and the environment, not only the test script.

In the last week of September, the open-source database project Apache Druid published its weekly triage of failing tests. Six of its seven code changes on 25 September had a failed test run. Of the 35 failures, 34 had one cause: the tests could not download an outside component in time. Not one of those red results said anything about the code being tested.

That is the real damage of flaky tests. Every red result that isn’t a real bug teaches your team that red can be ignored. After a while, nobody looks at the results at all.

What is a flaky test?

A flaky test is an automated test that sometimes passes and sometimes fails, without any change to the code or to the test itself. You run it twice on the same version and get two different answers. That makes it different from a broken test. A broken test fails every time, often because the application changed and the test was not updated. A flaky test is worse in one way: you can’t trust either answer. A green result might be luck. A red result might be noise. Take a test that logs in, adds a product to the basket and checks the total. It passes nine times out of ten. The tenth time, the basket page loads a little slower than usual, the test checks the total before it appears, and the run turns red. Nothing is wrong with the shop. Something is wrong with the test, or with the conditions it runs in.

What causes flaky tests?

Most flaky tests come from a handful of causes. The test assumes something about its surroundings that is only true most of the time. These are the usual ones:

• Timing. The test checks the screen or a response before the system is ready. A fixed pause of two seconds works on a quiet day and fails on a busy one.
• Test order. One test leaves something behind, like a logged-in user or a changed setting, and the next test depends on it without saying so.
• Outside services. The test calls a payment provider, a mail server or another team’s system. When that service is slow or down, your test fails.
• Parallel runs. Tests that run at the same time compete for the same account, record or file.
• Test data. The record the test needs has changed, expired or disappeared.

The first four get most of the attention online. The last one is the cause we run into most in business software.

Why is test data the cause you see most in business systems?

In a webshop, an ERP system or a CRM, tests don’t run in a clean lab. They run in a shared test environment, connected to other systems, used by other teams. Data in that environment changes all the time. Someone runs a manual test and closes the order your test expects to find. A nightly sync from another system overwrites the customer record. A discount code expires at the end of the month. A product goes out of stock because another test bought the last one. None of this is visible in the test script. The test simply finds a different world than the one it was written for, and fails. At e-commerce clients we see this pattern often. Tests fail regularly because nobody manages the test data, or because data in the connected systems keeps changing.

Connections to other systems add a second layer. In Salesforce projects we see tests fail now and then because an integration answers slowly or limits the number of requests. Those failures are hard to reproduce. By the time someone looks, the integration is fast again and the test passes.

How do you fix flaky tests for good?

Start by finding them. Look at the run history: which tests changed result without a code change in between? Most CI tools can show this per test. Tag those tests so they stand out. Then fix the cause, not the symptom.

• Wait for the right thing. Replace fixed pauses with checks that wait for a specific state, like a button that can be clicked or a response that has arrived. Tools such as Playwright wait for elements automatically before they act.
• Make every test stand on its own. Each test creates what it needs and cleans up after itself. No test depends on another one running first.
• Control your test data. This is what test data management means in plain words. You decide which data your tests use, instead of whatever happens to be in the environment. Build a known set of customers, products and orders before the run. Reset it when the run starts. Then the same test meets the same world every night.
• Replace outside services where the test isn’t about them. A test of your basket doesn’t need the real payment provider. A stand-in that always answers the same way removes a whole class of failures. Keep a few separate tests that do check the real connection.
• Quarantine with an owner and a date. Move a flaky test out of the main run so it stops blocking releases. Give it a name and a deadline, or it stays there forever.

Be careful with automatic retries. Running a failed test again until it passes keeps the pipeline green. It also hides the problem. Playwright marks a test that passed on retry as flaky, which helps, as long as someone reads that label.

What does a flaky test suite cost your team?

The direct cost is time. Someone reruns the build, reads logs and finds out it was nothing. The bigger cost is trust. When a red result usually means “try again”, people stop treating it as a signal. A real bug then gets the same treatment as the noise. We hear it in many conversations: a test breaks, it gets skipped “just for now”, and nobody comes back to it. A year later, nobody knows whether the coverage number can still be trusted. That is how a regression test suite stops working, quietly, one skipped test at a time.

Large companies take this seriously. Google has published research on finding the root cause of flaky tests in its own code base. For a smaller team the lesson is simpler: a test suite is only worth something if a red result means something.

What this means for your team

Pick your ten most important tests and check their history for the last month. Did any of them change result without a code change? If so, look at the data and the environment before you touch the script. Then decide who owns the nightly run. A flaky test that nobody owns never gets fixed.

And ownership is usually exactly what’s missing. Everyone relies on the test suite, but nobody has time to look after it. That is where we help teams. With Test Automation as a Service we run and maintain your automated regression tests alongside your team, including the test data they depend on. We cut manual regression testing by up to 70% for teams that still run their regression tests by hand. Your tests run every night, and we keep them reliable. You get a stable test suite your team can trust, one that doesn’t depend on a single person or add to your team’s workload. We build, document and maintain the suite, so it stays yours.

Frequently asked questions

Should you just rerun a flaky test?

Once, to confirm it is flaky, is fine. Rerunning until it passes is not a fix. It hides the cause, and real bugs that fail now and then get hidden the same way. Log every rerun, so you can see which tests need attention.

Are flaky tests a sign of bad test automation?

Not always. Every suite of some size has a few. A suite with many of them usually has the same few causes. Think of fixed pauses, tests that depend on each other and test data nobody controls. Those are design choices you can change.

How do you handle flaky tests in Playwright or Selenium?

The tool matters less than the cause. Playwright has automatic waiting and labels tests that pass on retry. With Selenium you add explicit waits yourself. In both, stable test data and independent tests do most of the work. Our comparison of Playwright and Selenium covers the other differences.

What is test data management?

Test data management means you decide and control the data your tests run on. You create a known set of records before a run and reset it afterwards. Tests then give the same result every time, as long as the software behaves the same.