In short: Regression testing means re-running tests after a change to check that what worked before still works. Run it after your own changes and after updates you don’t control. Automate the stable, high-value checks first. Most regression suites stall when nobody owns the nightly run, so plan for upkeep from day one.

On 8 September, Microsoft shipped its monthly Windows security update. Microsoft’s own support page now lists five known issues for that update. Remote Desktop connections dropped after a few minutes. On some computers, people could no longer sign in with their company account. Microsoft shipped fixes on 14 and 22 September, and at the end of the month one issue still had only a workaround.

Your own software runs on top of updates like this. That is what regression testing is for: proving that what worked yesterday still works today.

What is regression testing?

Regression testing is re-running existing tests after a change, to check that features which already worked still work. A regression is a bug in something that used to be fine. The change can be anything: a new feature, a bug fix, cleaned-up code, a new library version or an operating system update. The ISTQB is the international body for software testing. It describes regression testing as testing that looks for new defects in the parts of the software that did not change. So the test covers everything around the change. Picture an online shop that adds a new payment option. The regression test checks that login, search, the basket and the old payment options still behave. Retesting is something else. A retest checks that one specific bug is now fixed. A regression test checks that the fix didn’t break anything else.

A simple regression test might log in as a test customer, add two items to the basket, apply a discount code and check the total. If the total changes, someone needs to look.

When should you run regression tests?

Run regression tests after every change that could touch working features. That includes your own work: new features, bug fixes and changes to the code. It also includes changes you don’t make yourself. Operating systems and browsers update every month. Cloud platforms ship on their own schedule. Salesforce, for example, releases new features three times a year, in spring, summer and winter, and each one needs its own round of Salesforce release testing. Each of these updates can break a flow your team hasn’t touched in months. The September Windows update is an example: teams that shipped nothing that week could still see Remote Desktop fail. A useful rule: if anything in your stack changed, the regression run should happen before your users notice.

For most teams, that comes down to three moments:

• After every build or merge: a quick set of core checks.
• Before every release: the full suite.
• After updates you don’t control: operating system patches, browser versions, platform releases and library upgrades.

Should you automate regression testing?

Yes, for most of it. Regression tests repeat the same steps every time, and that is exactly the work computers do well. An automated suite can run every night while your team sleeps. It never skips a step because it’s Friday afternoon. Manual regression testing starts out fine, but it grows with every release. After a year, one manual round can take days of tester time. Automation has a cost too. Tests need upkeep when the application changes. Badly built tests fail for no clear reason. Those are called flaky tests, and they wear down trust fast. So keep people on the work that needs judgment: exploratory testing, new features, usability and anything visual that is hard to describe in a rule. The best set-up is a mix: the machine checks what must never break, and your testers look for what nobody has thought of yet.

A simple test for each case: automate a check when it runs often, the flow is stable and a failure would hurt. Keep it manual when the feature changes every sprint or the check runs once a year.

What should go into your first regression test suite?

Start small, and start with what hurts most. A first suite of 10 to 20 tests is enough if they cover the flows your business can’t live without. A small suite that runs every night and that everyone trusts is worth more than 500 tests nobody reads. Build it in this order:

1. List your key business flows, such as logging in, placing an order or approving a request. Rank them by the damage a failure would do.
2. Automate the top five first, from start to finish.
3. Add the bugs that have come back before, and from now on a test for every bug that reaches production.
4. Add the integrations that break most often, such as payment providers, login systems or data exports.
5. Run the suite on a schedule, so it never depends on someone remembering. Then grow it one flow at a time.

If you are still working out what to test at all, start with a test strategy before you automate anything.

Why do regression test suites stop working?

Most regression suites go through the same phase. A fast start, coverage climbs and the tests find real bugs. Then, a few months in, the growth stops. Fixing old tests takes up the time that used to go into new ones. Someone skips a broken test “just for now”, and nobody comes back to it. We hear a related story in conversations about test automation. “We already have that sorted” often means one person built most of it. Nobody else understands the structure, and maintenance was never planned in. When that person moves on, the suite still exists, but nobody runs it and nobody trusts it. The usual cause is ownership. Treat a regression suite like a small product. Someone needs to fix broken tests in the same week, remove tests that no longer matter and explain what the results mean.

What this means for your team

Start with three questions. Which flows must never break? Do you run a regression check after updates you didn’t make? And who fixes a broken test this week? If the answers are unclear, begin there before you buy another tool.

This is where Nekst fits. With Test Automation as a Service, we build and run your automated regression suite, working alongside your team. We run the regression suite, you decide what ships. We cut manual regression testing by up to 70%. Your tests run every night, and we keep them running. Not a project that stops when the person who built it moves on. Your testers get their time back for the work that needs them.

We start by going through your current testing with your team in the first days and finding the bottlenecks. Within the first month there is a working proof of concept. We build, document and maintain the suite, so it stays your asset.

That is what we did for MyTerminal, the online logistics platform of Hutchison Ports ECT Rotterdam. The platform grew its customer base by 400% in one year and released new features at a high pace. Ana Duško of MyTerminal put it like this: “Good test automation, set up by Nekst IT, has contributed to a continuous performance and a high speed of release.”

Frequently asked questions

What is the difference between regression testing and UAT?

User acceptance testing (UAT) checks whether a new feature does what the business asked for. Business users usually do it. Regression testing checks that existing features still work after a change. You need both: UAT for what is new, regression testing for what already worked.

Is regression testing the same as functional testing?

No. Functional testing checks that a feature works as specified, often for the first time. Regression testing repeats checks, often functional ones, after something changed. Many regression tests are functional tests that you keep and run again.

How often should you run regression tests?

Run a small core set on every build and the full suite before every release. Also run it after updates outside your control, such as operating system patches or platform releases. A nightly run is a good default for most teams.

Can you automate all regression testing?

Not all of it, and you shouldn’t try. Automate the checks that are stable, repeated and worth the effort. Keep exploratory testing and visual judgment with people. Many teams end up with a large automated core and a short manual check.