Skip to content

What is test automation? A guide from the founders of Codeception and CodeceptJS

Mykhailo Poliarush6 min read

What is test automation? It is the use of software to run your tests and check the results, so every change to your product can be verified automatically, as often as you need. The International Software Testing Qualifications Board (ISTQB) defines it as "the use of software to perform or support test activities, e.g., test management, test design, test execution and results checking" (ISTQB Glossary v3.2).

What is test automation, and what does it replace?

It replaces the repetitive part of testing, not the thinking part. A person still decides what the product must do. Automation turns those decisions into executable checks: log in, add an item to the cart, call the payment API, compare the result with what was expected. Once a check is written, it runs on every commit, at night, in parallel and without getting bored.

What it does not replace is judgment. Exploratory testing, usability review and deciding which risks matter stay with people. In practice the two work together: people find new problems, and automation makes sure old problems never come back.

How does test automation work?

A typical setup has four parts:

  1. Tests, written as code or as readable scenarios, that drive the product and assert expected results.
  2. A framework that runs those tests, talks to browsers, APIs or databases, and reports what failed.
  3. A pipeline (CI) that runs the right tests on every change and blocks a merge when they fail.
  4. Test data and environments that make each run repeatable.

Tests live at different levels. A common model is the test pyramid, which Mike Cohn introduced in his book Succeeding with Agile. Ham Vocke's practical guide to the test pyramid on martinfowler.com sums up the rule: "Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end."

The test pyramid: how much to automate at each level

The test pyramid: how much to automate at each level
CriterionWhat it checksHow many tests
Unit testsOne class or function, in isolationLots: small and fast
Service tests (API, integration)Several parts working togetherSome: more coarse-grained
User interface and end-to-end testsWhole user journeys through the productVery few

Source: Ham Vocke, The Practical Test Pyramid, martinfowler.com (2018), after Mike Cohn

The reason is speed and stability. Vocke notes that end-to-end tests "are notoriously flaky and often fail for unexpected and unforeseeable reasons." Unit and API tests are cheap to run thousands of times a day; browser tests are worth keeping for the few user journeys that make you money.

What is a test automation framework?

A test automation framework is the structure your tests are built on. The ISTQB Test Automation Engineering syllabus puts it this way: "The TAF is the foundation of a TAS. It often includes a test harness, also known as test runner, and test libraries, test scripts and test suites" (ISTQB CTAL-TAE v2.0, 2024). TAS stands for test automation solution, the whole system of tools and tests. The same syllabus recommends keeping the number of framework layers low, because each extra layer adds design complexity.

In practice, a framework gives you a way to write tests, helpers to drive a browser or call an API, fixtures for test data, and reporting. Our team created two open-source frameworks: Codeception for PHP, whose GitHub repository dates from 2011 and has 4,852 stars, and CodeceptJS for JavaScript, whose repository dates from 2015 and has 4,253 stars (both checked on 18 November 2026). Both let you write tests as readable scenarios, so testers, developers and product owners can review the same test.

What is a test automation strategy?

A test automation strategy is the plan that decides what to automate, with which tools, by whom, and how you will know it pays off. The ISTQB Test Automation Strategy syllabus (2024, by Andrew Pollner and co-authors) lists what a strategy has to address: the intended purpose, risks, scope, stakeholders, tool selection, the test automation architecture and the environments. It also lists typical objectives, such as broader coverage, reduced test execution time, increased test frequency and reduced total cost and time to market.

The syllabus adds a condition that many teams skip: the strategy "needs to be applicable and customizable; its goals need to be achievable within time and cost constraints, and it needs to be kept up-to-date." A strategy written once for a vendor selection and never revisited is a document, not a strategy.

How do you start test automation? A step-by-step checklist

  1. Pick the risk first. List the user journeys and integrations where a bug costs the most: sign-up, checkout, billing, data exports.
  2. Agree what "correct" means. Write acceptance criteria with product owners before anyone writes a test.
  3. Choose a framework that fits your stack and your people. A PHP team and a JavaScript team need different tools; readable tests matter if non-developers will review them.
  4. Start at the API and unit level. Add a small number of end-to-end tests for critical journeys only.
  5. Run everything in CI. Fast tests on every commit, the full suite before release, no merge on a red build.
  6. Track flakiness from day one. Quarantine unstable tests with an owner and a deadline.
  7. Review the suite every quarter. Delete tests that never fail and never protect anything; add tests for every escaped bug.

What does test automation cost?

We will not quote a price, because no published benchmark fits every product. The cost depends on factors the ISTQB Test Automation Strategy syllabus names directly: time constraints, the expertise and number of automation engineers, test hardware, tool licenses, adaptability, maintenance, platform support (web, desktop, mobile), CI/CD support and test reporting. It calls the timeline "the first and most important cost factor."

Maintenance is the cost buyers underestimate. Unstable tests waste real engineering time: at Atlassian, flaky tests caused as much as 21% of master build failures in the Jira Frontend repository, and reruns waste over 150,000 hours of developer time each year, according to Atlassian Engineering (December 2025). The good news is that flakiness responds to management.

GitHub monolith: commits with a red build caused by a flaky test

Before flaky-test management (1 in 11)
9%
After (1 in 200)
0.5%

Source: GitHub Engineering, Reducing flaky builds by 18x (December 2020)

GitHub reported in 2020 that 1 in 11 commits in its monolith had at least one red build caused by a flaky test; after it introduced a system to manage flaky tests, that fell to 1 in 200, an 18x improvement.

The other side of the ledger is the cost of not testing. The Consortium for Information and Software Quality estimated the cost of poor software quality in the US at at least $2.41 trillion in its 2022 report. Treat it as an order of magnitude, not a precise figure.

Common test automation mistakes

  • The ice-cream cone. Most tests run through the user interface and very few at the unit or API level. Vocke warns this ends up "a nightmare to maintain" and slow to run.
  • Automating everything. Automating a test that never fails and protects nothing still costs maintenance. Automate by risk.
  • Ignoring flaky tests. A suite that fails at random teaches people to ignore red builds. Quarantine flaky tests and fix the cause, usually timing, shared state, test data or the environment. On Bitrise's mobile CI platform, the share of teams with any flaky test grew from 10% in 2022 to 26% in 2025 (Bitrise Mobile Insights 2025; vendor data).
  • Letting AI write tests nobody reviews. AI is good at drafting tests: 62% of respondents who create test cases use AI to help, according to DORA 2025. But the same report finds AI adoption still increases delivery instability, and 44.1% of developers in the Stack Overflow Developer Survey 2025 said they do not plan to use AI for testing code. A generated test that asserts the wrong thing is worse than no test, because it gives false confidence.
  • No owner. Test automation is software. Without someone accountable for it, it rots like any other code.

How SDCLabs can help with test automation

We built the frameworks other teams use. SDCLabs, founded in 2011 in Krakow, created Codeception and CodeceptJS, and our senior engineers run AI-assisted workflows: AI agents draft test cases, page objects and test data; engineers review them, keep only the meaningful ones and decide what goes into the suite; automated tests verify every change.

We can audit an existing suite, write a test automation strategy with you, build the first suites at the right levels, or join your team as QA engineers. See QA and test automation, the Codeception and CodeceptJS pages, and our trainings if you want your own team to own the suite. Engagements start in 1 to 2 weeks. Tell us about your product.

For where testing meets AI products, read what is AI testing; for the pipeline side, see DevOps and TestOps.

Sources checked on 2026-11-18.

Frequently asked questions

What is a test automation framework?

It is the structure your automated tests are built on: a test runner, libraries, test scripts and test suites, per the ISTQB Test Automation Engineering syllabus. Codeception and CodeceptJS are examples.

What is a test automation strategy?

The plan for what to automate, with which tools, by whom and how you measure the payoff. The ISTQB Test Automation Strategy syllabus says it must cover purpose, risks, scope, stakeholders, tools, architecture and environments, and be kept up to date.

Why do test automation tests fail?

Real bugs, or flakiness: tests that fail at random, usually because of timing, shared state, test data or the environment. End-to-end tests are notoriously flaky, which is one reason the test pyramid keeps them few.

What is the best platform for test automation?

The one that fits your stack and the people who will review the tests. For PHP there is Codeception, for JavaScript CodeceptJS; both are open source and were created by SDCLabs.

Can AI write automated tests?

It can draft them: 62% of respondents who create test cases use AI to help, per DORA 2025. An engineer still has to check that each test asserts the right behavior.

Related posts

Get an estimate for your product.