What is regression testing?
Definition
Regression testing is re-running existing tests after a change to confirm that features which worked before still work. It applies to bug fixes, new features, refactors, dependency upgrades and configuration changes. Most teams automate it: a fast selection runs on every commit, and the full suite runs before each release.
How does regression testing work?
Every change is a risk to code nobody touched. A bug fix in the cart can break invoices; a library upgrade can break login. Regression testing is the routine that catches those side effects. The ISTQB Foundation syllabus puts it this way: "Regression testing confirms that no adverse consequences have been caused by a change, including a fix that has already been confirmation tested" (ISTQB CTFL v4.0.1, September 2024, still the current Foundation syllabus).
A regression cycle has six steps:
- A change lands. New code, a fix, a dependency bump, a config edit or a new environment.
- Impact analysis. Work out which parts of the system the change could affect. ISTQB advises doing this first, to see how far the checks need to reach.
- Selection. Pick the tests that cover the affected areas, or run them all.
- Execution. Run the selection, usually in CI.
- Triage. Every red test is one of three things: a real regression, an intended behavior change that needs a test update, or a flaky test.
- Pinning. When a bug escapes to production, write a test that reproduces it before the fix ships. That test joins the suite for good.
The scope is wider than the code you changed. ISTQB notes the adverse effects can land in the same component, in other components, or in "other connected systems", and that the checks "can also be related to the environment".
What is the difference between regression testing and retesting?
Retesting is the everyday name for what ISTQB calls confirmation testing. They answer different questions and usually run back to back.
| Confirmation testing (retesting) | Regression testing | Testing a new feature | |
|---|---|---|---|
| Question | Is this defect really fixed? | Did the change break anything that used to work? | Does the new behavior match the requirement? |
| Scope | The steps that reproduced the defect | Unchanged areas around and beyond the change | The new code paths |
| Tests used | The failed tests, plus new ones for the fix | The existing suite, or a selection of it | Newly written tests |
| Repeats | Once per fix | On every change, for the life of the product | Until the feature ships, then they join the regression suite |
The last row explains why regression suites only grow. ISTQB says so directly: the number of regression test cases "will increase with each iteration or release, so regression testing is a strong candidate for automation."
Which tests belong in a regression run?
Running everything on every commit stops scaling once the suite takes longer than a coffee break. A survey by Yoo and Harman (published 2012) groups the alternatives into three techniques: minimization removes redundant tests, selection picks the tests relevant to recent changes, and prioritization orders tests so faults show up early.
| Technique | What it does | When it fits |
|---|---|---|
| Retest all | Runs the full suite | Before a release, on the main branch, after large upgrades |
| Selection | Runs only the tests relevant to the change | Pull requests and pre-merge checks |
| Prioritization | Orders tests so likely failures run first | Long suites where early feedback matters |
| Minimization | Removes redundant tests | Suites that grew by copy and paste |
For prioritization, ISTQB lists three common strategies: risk-based (tests for the biggest risks first), coverage-based (tests with the highest code coverage first) and requirements-based (tests for the requirements stakeholders rank highest).
How do you automate regression testing?
You need three things in the test runner: a way to mark tests, a way to link a test to the bug it pins, and a way to run only what a change affects. Playwright Test, the open-source runner from Microsoft, has all three. Tests take tags such as @regression and annotations of type issue that show up in the report (Playwright annotations). The --only-changed option, added in version 1.46, runs the test files changed since a git ref. Per the Playwright release notes, "This will also run all test files that import any changed files."
Here is tests/cart.spec.js, with a test pinned to an old cart bug, where a coupon was applied to each line and then again to the order total:
import { test, expect } from '@playwright/test'
import { total } from '../src/cart.js'
test('sums price times quantity', () => {
expect(total([{ price: 500, qty: 2 }, { price: 250, qty: 1 }])).toBe(1250)
})
// Pinned to a fixed bug: the coupon was once applied to each line and again to the total.
test('applies a coupon once per order', {
tag: '@regression',
annotation: { type: 'issue', description: 'SHOP-142' },
}, () => {
expect(total([{ price: 1000, qty: 1 }, { price: 1000, qty: 1 }], 10)).toBe(1800)
})
We committed this suite to a fresh git repository with a second, unrelated spec for password rules. Then a "refactor" in src/cart.js moved the discount into each line and brought the old bug back, without touching any test file. Run on 2026-11-19 with @playwright/test 1.65.1 on Node.js 22.23.5 (excerpt; the code frame and file paths are cut):
$ npx playwright test --only-changed
Running 2 tests using 1 worker
✓ 1 tests/cart.spec.js:4:1 › sums price times quantity (3ms)
✘ 2 tests/cart.spec.js:9:1 › applies a coupon once per order @regression (3ms)
1) tests/cart.spec.js:9:1 › applies a coupon once per order @regression
Error: expect(received).toBe(expected) // Object.is equality
Expected: 1800
Received: 1620
1 failed
tests/cart.spec.js:9:1 › applies a coupon once per order @regression
1 passed (394ms)
Only the cart spec ran, because it imports the changed module; the password spec was skipped. The pinned test caught the regression, and npx playwright test --grep @regression runs the pinned set alone. One caveat from the same release notes: under Bun, --only-changed does not follow imports.
How often should you run regression tests?
Match each layer of the suite to a trigger:
- Every pull request: the affected selection plus fast unit and API tests.
- Every merge to main: the full fast suite, because selection by imports cannot see config files, database migrations or environment changes.
- Before each release and after dependency upgrades: everything, including slow end-to-end tests.
To check whether this works, watch outcomes rather than test counts. DORA's change fail rate is "the ratio of deployments that require immediate intervention following a deployment" (DORA metrics guide). If it climbs while the suite stays green, regressions are escaping, and each escaped bug should end up as a new pinned test.
Why does a regression suite stop catching bugs?
- The tests wear out. ISTQB lists it among its seven testing principles: tests repeated many times "become increasingly ineffective in detecting new defects". Add tests for every escaped bug and retire the ones that never fail and protect nothing.
- Red becomes normal. When people rerun until green, a real regression looks like noise. Our test automation guide covers how teams quarantine and fix flaky tests.
- Expected values get edited to match. A failing assertion updated without a look at why it failed turns a regression into the new spec.
- Selection gets trusted blindly. Import graphs miss runtime wiring, feature flags and shared data. Keep a full run somewhere in the pipeline.
Our QA and test automation team builds and audits regression suites in Codeception, CodeceptJS and Playwright, decides what to automate at which level, and wires the suites into your CI.
Sources
- Certified Tester Foundation Level Syllabus v4.0.1, ISTQB, 2024-09-15
- Regression testing minimization, selection and prioritization: a survey, Software Testing, Verification and Reliability (Yoo, Harman), 2012-02-23
- Annotations, Playwright docs (Microsoft), 2026-11
- Release notes, Playwright docs (Microsoft), 2026-11
- DORA's software delivery performance metrics, DORA (Google Cloud), 2026-01-05
Frequently asked questions
Who does regression testing?
Usually nobody by hand. Developers and QA engineers write the tests, and CI runs them on every change. People step in to triage failures and to explore areas that automated checks do not reach.
What is regression testing in agile?
In agile teams it runs every sprint, often every commit, because each increment changes working software. That pace is why agile teams automate most regression checks and keep only exploratory sessions manual.
Is regression testing the same as smoke testing?
No. A smoke test is a short check that a build starts and its main paths respond. A regression run goes deeper and checks that existing behavior, including old bug fixes, still holds.
Can regression testing be fully automated?
Most of it can, and should be, because the same checks repeat on every change. Visual polish, usability and unexpected side effects in new areas still need a person.
Services
Read next
Related terms
Mikhail BodnarchukFounder of Codeception and CodeceptJSBoth are open-source testing frameworks: Codeception for PHP, CodeceptJS for JavaScript.LinkedInGitHub