70% on one benchmark, 23% on the next: how to benchmark AI agents
In 2026, benchmarking an AI agent means proving that it works on your tasks, on every run, at a cost you can accept. Public scores no longer prove that. Top models that scored over 70% on SWE-bench Verified resolved about 23% of tasks on the harder SWE-Bench Pro, according to Scale AI's SWE-Bench Pro paper. Teams that pick agents from leaderboards are measuring the wrong thing.
Are public agent benchmarks still useful?
They show the direction of progress, but they wear out fast. The Stanford AI Index 2026 puts it bluntly: "Evaluations intended to be challenging for years are saturated in months." Accuracy on Humanity's Last Exam went from under 10% to 38.3% in a single year, and on Terminal-Bench 2.0, which tests agents in real terminal environments, the top score rose from 20% in February 2025 to 77.3% in early 2026, according to the same report.
Benchmarks saturate fast: best reported scores
| Criterion | Benchmark | What it tests | Earlier | Latest |
|---|---|---|---|---|
| Humanity's Last Exam | 2,700 expert-level questions | Under 10% (2024) | 38.3% (2025) | |
| OSWorld | Real computer tasks (students: about 72%) | About 12% | 66.3% | |
| Terminal-Bench 2.0 | Tasks in real terminal environments | 20% (Feb 2025) | 77.3% (early 2026) |
Source: Stanford HAI, AI Index Report 2026, pp.72, 95, 101, 113 (April 2026)
Once the top models cluster near the ceiling, a benchmark stops telling them apart. A score that is a few points higher tells you little about your own codebase, your tools or your customers.
Why can't you trust a leaderboard score?
Because the test may be broken, or the model may have seen it. In February 2026, OpenAI stopped reporting SWE-bench Verified. Its audit of 138 problems that its o3 model did not consistently solve over 64 runs found that 59.4% of them had material issues in test design or problem description. It also found that every frontier model it tested could reproduce the original human-written fix for some tasks, a sign that the problems had leaked into training data. OpenAI is a model vendor, but its conclusion matches independent work.
The AI Index 2026 reports that contamination "can lead to falsely inflated scores", and cites an audit that found invalid questions in nine widely used benchmarks, from 2% on MMLU Math to 42% on GSM8K. Scale AI's own analysis found that 161 of the 500 SWE-bench Verified problems need only a one- or two-line change, according to the SWE-Bench Pro paper.
Does an agent that passes once pass every time?
Often it does not, and most leaderboards do not measure this. The τ-bench paper introduced pass^k, the chance that an agent solves the same task in all k attempts. In the 2024 paper, gpt-4o solved under 50% of its tasks and its pass^8 in the retail domain was under 25%. A customer who asks the same question eight times should not get eight different outcomes.
The ceiling is still low on this kind of task. Leading models score between 62.9% and 70.2% pass@1 on τ-bench, and no model exceeds 71%, according to the AI Index 2026. That is a single attempt; the share of tasks solved on every attempt cannot be higher.
What do cost and latency add to the picture?
They decide whether an agent is usable. Researchers behind the Holistic Agent Leaderboard ran 21,730 agent rollouts across 9 models and 9 benchmarks for a total cost of about $40,000. They found that higher reasoning effort reduced accuracy in the majority of runs, so paying for more thinking does not guarantee better results. Their log review also caught agents searching for the benchmark on HuggingFace instead of solving the task, and misusing credit cards in flight booking tasks. A single accuracy number hides all of that.
An earlier paper by some of the same researchers, AI Agents That Matter, found that a narrow focus on accuracy made state-of-the-art agents "needlessly complex and costly", and that many agent benchmarks have inadequate holdout sets, or none at all. In practice, latency matters too: in LangChain's State of Agent Engineering survey of 1,340 practitioners, latency was the second biggest challenge (20%), after quality.
Why build a task-specific benchmark?
Because the way a task is described changes the result as much as the model does. In the SWE-Bench Pro paper, GPT-5 (high) resolved 25.9% of public tasks when given human-written requirements and an interface spec and 8.4% with the problem statement alone, while Claude Opus 4.1 went from 22.7% to 8.2%.
SWE-Bench Pro resolve rate, with and without human-written requirements
- GPT-5 (high), full spec
- 25.9%
- GPT-5 (high), problem statement only
- 8.4%
- Claude Opus 4.1, full spec
- 22.7%
- Claude Opus 4.1, problem statement only
- 8.2%
Source: Scale AI, SWE-Bench Pro, arXiv 2509.16941v2, Table 3 (public set, Nov 2025)
Private code is harder still. On SWE-Bench Pro's commercial set, built from 276 problems in proprietary startup repositories, the best result in the paper was 17.8% (Claude Opus 4.1). Scale AI sells evaluation services, so read its numbers as one vendor's benchmark, but the lesson is general: your repositories, your tickets and your specs are the only test set that predicts how an agent will do for you.
Once an agent is live, the benchmark moves into production, and teams start with tracing. In the LangChain survey, 62% of respondents have detailed tracing of individual agent steps and tool calls, rising to 71.5% among teams with agents in production. Online evaluations, which score live traffic, are run by 37.3% of all respondents and 44.8% of teams in production, according to the same survey. LangChain notes that cost concerns dropped from the previous year. LangChain sells agent tooling and surveyed its own audience, so these figures describe advanced teams.
What does a good agent benchmark look like?
A useful agent benchmark is small, private and repeatable. A practical way to start:
- Sample tasks from your own work. Take a representative set of real tickets, conversations or workflows with known good outcomes, and keep part of them as a holdout that nobody tunes against.
- Grade outcomes, not text. Check the final state, such as the database row, the passing test or the refund issued, the way τ-bench and SWE-bench do.
- Run every task several times. Report pass@1 and pass^k side by side, so you see both capability and consistency.
- Record cost and latency per task. Compare agents on a cost and accuracy chart, not on accuracy alone.
- Read the logs. Sample failed and passed runs. Shortcuts, leaked answers and unsafe tool calls show up in traces, not in scores.
- Rerun on every change. A new model, prompt or tool is a new agent. Keep the benchmark in CI and compare against the last baseline.
Ready to benchmark your agents?
If you are choosing between models, frameworks or vendors, or you need to know whether last week's prompt change made your agent better or worse, SDCLabs' AI agent benchmarking service is the place to start. We founded the open-source testing frameworks Codeception and CodeceptJS, and our senior engineers stay in control of AI-assisted work with automated tests on every change. Engagements start in 1 to 2 weeks, and we reply within 2 hours on business days. Send us your use case.
Testing an AI product end to end? See AI testing services and our state of AI engineering in 2026 report.
Sources checked on 2026-11-17.
Frequently asked questions
Why did OpenAI stop reporting SWE-bench Verified?
OpenAI audited 138 problems that o3 did not consistently solve and found that 59.4% had material issues in test design or problem description, and that frontier models had seen some problems during training. See OpenAI's note.
What is pass^k in agent benchmarking?
pass^k is the chance that an agent solves the same task in all k attempts, introduced in the τ-bench paper. It measures consistency, which a single pass@1 score hides.
Do cost and latency matter when comparing agents?
Yes. The Holistic Agent Leaderboard found that higher reasoning effort reduced accuracy in the majority of runs, and latency was the second biggest challenge (20%) in LangChain's survey.
How do you build a task-specific agent benchmark?
Sample real tasks with known good outcomes, keep a holdout set, grade final outcomes, run each task several times, record cost and latency, read the logs and rerun on every model or prompt change.
