Skip to content

An agent that passes once can fail twice: a practical guide to AI agent testing

Mykhailo Poliarush7 min read

AI agent testing checks whether an agent completes real tasks in a realistic environment: the right end state, correct tool calls, consistent results across repeated runs, acceptable cost and latency, and safe behavior when inputs are hostile. It differs from testing a chatbot because an agent acts. It changes records, sends messages and spends money, so you test what it did, not only what it said.

How is AI agent testing different from testing a chatbot?

A chatbot test grades a reply. An agent test grades a whole run. Anthropic's eval guide uses a precise vocabulary that is worth adopting (Anthropic, January 2026):

  • Task: one test with defined inputs and success criteria.
  • Trial: one attempt at a task. You run several, because outputs vary between runs.
  • Grader: logic that scores one aspect of a run.
  • Transcript: the full record of messages, tool calls and intermediate results.
  • Outcome: the final state of the environment.

The difference between transcript and outcome is the heart of agent testing. A flight-booking agent can say "your flight has been booked"; the outcome is whether a reservation exists in the database. Your tests need access to that database, in a sandbox.

What goes into an agent task suite?

Start with 20 to 50 tasks drawn from real failures, which Anthropic calls a great start (Anthropic). Take them from support tickets, bug reports and the checks your team runs by hand. For each task:

  • Write a reference solution that passes every grader. It proves the task is solvable. If an agent scores 0% over many trials, the task is usually broken, not the agent (Anthropic).
  • Cover both directions: cases where the agent should act and cases where it should refuse, ask or hand over.
  • Isolate every trial in a clean environment. Shared state skews results: in Anthropic's own evals, Claude gained an unfair advantage by reading the git history of earlier trials.
  • Simulate the user carefully. Conversational agents need a second model playing the customer, and that model makes mistakes too. In tau2-bench, the simulated user made task-critical errors in 13% of airline conversations and 6% of telecom ones (tau2-bench).

Keep two suites. Capability tasks show what the agent cannot do yet. Regression tasks should pass nearly every time and block releases when they fail. For dataset basics, see what is AI testing?

How do you test tool-call correctness?

Check the calls that matter, not the exact path. Anthropic's example refund task requires an identity check, a refund call with an amount of at most 100 and a confirmation, then verifies the ticket and refund status in the database (Anthropic). It also warns that asserting a fixed sequence of tool calls is too rigid, because agents find valid routes the test author did not expect. Grade what the agent produced, give partial credit for multi-step tasks, and keep strict assertions for forbidden calls and high-risk parameters.

Watch for the classic tool failures. In Agent-SafetyBench, a 2,000-case benchmark from Tsinghua University, agents often fabricated parameters when information was missing and ignored the dangers of harmful tools (Agent-SafetyBench). Log every tool call with its runtime, the number of calls, token use and tool errors, as Anthropic does when it evaluates tools.

How do you measure reliability across runs?

Run every task several times and report two numbers. pass@k is the chance that at least one of k runs succeeds. pass^k is the chance that all k succeed, which is what a customer-facing agent needs. An agent that succeeds 75% of the time per run passes three runs in a row only about 42% of the time (Anthropic). We cover how public benchmarks report this in how to benchmark AI agents.

Reliability also depends on the domain, so test on your own tasks. In tau2-bench, built by Sierra, the same gpt-4.1 agent succeeded on the first try in 74% of retail tasks, 56% of airline tasks and 34% of telecom tasks (tau2-bench).

Same agent, different domain: gpt-4.1 first-try success rate (pass^1)

Retail
74%
Airline
56%
Telecom
34%

Source: Barres et al. (Sierra), tau2-bench, arXiv 2506.07982 (June 2025; each task run four times)

The same paper found success falling close to zero for tasks that needed more than 7 actions, and a drop of 18% for gpt-4.1 when the agent had to guide a user instead of acting alone (tau2-bench). Long tasks and shared control deserve their own test cases.

How do you test cost and latency?

Treat them as pass criteria, not afterthoughts. Track turns, tool calls, total tokens, time to first token and time to last token for every trial (Anthropic). Latency is the second biggest challenge reported by agent builders, at 20%, in LangChain's vendor survey (LangChain).

More effort is not always better. Across 21,730 agent runs costing about $40,000, the Holistic Agent Leaderboard found that higher reasoning effort reduced accuracy in the majority of runs (HAL). Set a budget per task, and fail the test when the agent exceeds it.

How do you test AI agent safety?

Test what the agent does, not only what it says. Agent-SafetyBench ran 16 agents built on 2024 models through 349 environments and found none with a safety score above 60%. On average, agents were far safer in cases without tool environments than in cases where they had tools to use (Agent-SafetyBench).

Agents are safer in what they say than in what they do: average safety score of 16 agents

Cases without environments (content safety)
68.4%
All cases (total)
38.5%
Cases with tool environments (behavior safety)
30.4%

Source: Zhang et al. (Tsinghua University), Agent-SafetyBench, Table 5, arXiv 2412.14470 (2,000 test cases, 2024 models)

The same study found that defense prompts alone gave limited gains. Build attack cases into the suite instead:

  • Prompt injection through tool outputs: emails, web pages and documents that try to redirect the agent. AgentDojo offers 97 tasks and 629 security test cases for exactly this.
  • Goal hijack and tool misuse, the first two risks on the OWASP Top 10 for Agentic Applications.
  • Permission checks: confirm that the agent cannot reach tools and data outside its role, even when asked.
  • Escalation: confirm it hands high-risk actions to a human.

Who checks the graders?

You do, by reading transcripts. Anthropic's Opus 4.5 first scored 42% on CORE-Bench; after the team fixed rigid grading and ambiguous tasks and used a less constrained scaffold, it scored 95% (Anthropic). Logs also catch cheating: HAL found agents searching for the benchmark on HuggingFace instead of solving tasks (HAL). Model graders need calibration too: used directly as a safety grader, GPT-4o was right 75.5% of the time (Agent-SafetyBench, 2024 models). We cover judge bias in LLM evaluation.

AI agent testing step by step

  1. Define success per task: end state, allowed and forbidden tool calls, budget for turns, tokens and time.
  2. Build 20 to 50 tasks from real cases, both directions, each with a reference solution.
  3. Stand up a sandbox with test data, mocked or staging systems and a clean reset per trial.
  4. Write graders: state checks first, tool-call assertions for risky actions, calibrated model graders for tone and reasoning.
  5. Run each task several times and report pass^k, cost and latency.
  6. Add attack and permission cases.
  7. Read failing transcripts and fix the task, the grader or the agent.
  8. Gate releases in CI on the regression suite, and add every production failure as a new task.

In-house or with a partner?

  • In-house works when you have engineers who know both test automation and LLM behavior, plus time to build a harness and sandbox.
  • A partner helps when you need a first suite in weeks, an independent view of the graders, or a benchmark to choose between models and frameworks.
  • Either way, the product team owns the definition of success, and the suite stays in your repository.

What does AI agent testing cost?

We have not found verified price ranges, so these are the drivers: number and complexity of tasks, how much of your stack the sandbox must reproduce, runs per task, model graders and simulated users, human review, and the risk level of the agent's actions. Run costs are measurable: tau2-bench reports about $0.086 per task for the agent and $0.059 for the simulated user with gpt-4.1 (tau2-bench).

Common mistakes in AI agent testing

  • Grading the final message instead of the end state.
  • One run per task. Reliability needs repeats.
  • Asserting exact tool sequences, then deleting tests when they break.
  • Trusting the grader without reading transcripts.
  • No hostile inputs. Agents read untrusted content all day.

How SDCLabs can help

We founded the open-source testing frameworks Codeception and CodeceptJS, and we test agents the way we test software: in CI, with reviewed graders. AI agent benchmarking builds the task suite and harness and measures success rate, cost, latency, tool use and reliability across runs on your real work. AI testing services add evaluation datasets, red-team runs and regression gates for the LLM features around the agent, and QA and test automation covers the rest of the product in the same pipeline. Still deciding what to automate? Start with an AI readiness assessment.

Engagements start in 1 to 2 weeks. Tell us about the agent you need to test.

Sources checked on 2026-11-18.

Frequently asked questions

How do you test an AI agent?

Build a suite of real tasks in a sandbox, grade the end state and the risky tool calls, run each task several times, track cost and latency, add attack cases and gate releases in CI on the results.

How many test cases do you need to start testing an AI agent?

Anthropic calls 20 to 50 simple tasks drawn from real failures a great start (Anthropic). Grow the suite with every production failure.

What is pass^k in AI agent testing?

The probability that all k runs of a task succeed. An agent with a 75% success rate per run passes three runs in a row only about 42% of the time (Anthropic).

Should AI agent tests check every tool call?

Check the outcome, the required and forbidden calls and risky parameters, but not an exact sequence. Anthropic found fixed-sequence checks too rigid because agents find valid routes testers did not expect (Anthropic).

How do you test AI agent safety?

Add prompt injection through tool outputs, permission and escalation cases to the suite. In Agent-SafetyBench, none of 16 agents scored above 60% on safety, and defense prompts alone gave limited gains.

Related posts

Get an estimate for your product.