Is your LLM evaluation measuring your app or your judge?
LLM evaluation is how you prove that an application built on a large language model gives correct, safe and useful answers on your own tasks: you run a fixed set of real cases, score every answer with code checks, model graders or people, and compare the scores on every change. It tells you before release whether a new prompt, model or retrieval setup made the product better or worse, as long as you also test the graders.
What is LLM evaluation, and how is it different from a leaderboard?
Public leaderboards rank models in general. Your users meet a system: your prompts, your documents, your tools and your rules. LLM evaluation measures that system on the questions it will actually get.
Leaderboards are useful for a shortlist, not a decision. Arena (formerly LMArena, and before that Chatbot Arena) ranks models from crowdsourced pairwise votes; its 2024 paper reported over 240K votes. A 2025 study, The Leaderboard Illusion, found that providers could test private variants before release and publish the best one, and it counted 27 private variants tested by Meta before Llama 4. Its authors are mostly at Cohere Labs, which also submits models to the Arena. Either way, a top rank does not say how a model handles your refund policy.
Offline vs online evals: what is the difference?
You need both, at different times.
- Offline evals run a fixed dataset before release, on every prompt, model or retrieval change, usually in CI. They are reproducible and touch no users, but they only cover what you thought to include.
- Online evals score live traffic after launch: sampled answers graded automatically, user feedback, A/B tests and manual review of transcripts. They catch what the dataset missed, but problems reach users first.
Anthropic's guide to evals recommends automated evals pre-launch and in CI/CD, production monitoring after launch to detect drift, and human studies to calibrate model graders (Anthropic, January 2026). Neither half is universal yet: in LangChain's vendor survey, 52.4% of respondents run offline evaluations and 37.3% run online ones (LangChain).
Which metrics should an LLM evaluation track?
Pick metrics per behavior, not one overall score.
- Format and rules: valid JSON, required fields, citations present, forbidden content absent. Code checks these exactly and cheaply.
- Correctness: exact match where an answer is unique, a rubric or a reference answer where it is not.
- Retrieval quality (RAG): whether the retrieved passages are relevant, whether the answer stays faithful to them, and the quality of the answer itself. These are the three dimensions Ragas was built to score without human labels.
- Safety: refusals, data leaks and prompt injection, which OWASP ranks as risk number one for LLM applications (OWASP).
- Operations: latency, tokens and cost per task, tracked on the same dataset so a "better" prompt that doubles cost is visible.
Split the suite in two. Capability evals start with a low pass rate and show progress. Regression evals should pass nearly 100% of the time, so any drop means something broke (Anthropic).
How do you build an evaluation dataset?
Start small and real. Anthropic's engineers write that 20 to 50 simple tasks drawn from real failures is a great start, because early changes have large effects (Anthropic). Good sources are support tickets, bug reports and the checks you already run by hand before a release.
Three rules keep the dataset honest:
- Unambiguous cases. A good task is one where two domain experts would independently reach the same verdict. If they would not, fix the case, not the model.
- Both directions. Test where a behavior should happen and where it should not, or you will tune an assistant that, for example, searches for everything.
- A held-out set. Keep cases you never tune against, so you do not overfit to your own tests, as Anthropic did with its tool evaluations.
We describe the golden dataset in more detail in what is AI testing?
Can you trust LLM-as-a-judge?
Partly, after calibration. Model graders scale to open-ended answers that code cannot check, and they can agree with people well: in the 2023 MT-bench study, GPT-4 agreed with human experts on 85% of non-tie votes, while humans agreed with each other on 81% (Zheng et al.). The same study measured the biases you have to design around.
How 2023 LLM judges handled two known biases
| Criterion | Judge (2023 model) | Same verdict after answer order is swapped | Preferred a padded, repetitive answer |
|---|---|---|---|
| GPT-4 | 65.0% | 8.7% | |
| GPT-3.5 | 46.2% | 91.3% | |
| Claude-v1 | 23.8% | 91.3% |
- Position bias. With two similar answers, GPT-4 gave the same verdict after the order was swapped in only 65.0% of cases, and Claude-v1 in 23.8% (Zheng et al.). In another study, simply reordering answers let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the judge (Wang et al.).
- Verbosity bias. Padding a list with rephrased items fooled GPT-3.5 and Claude-v1 in 91.3% of 23 test answers, GPT-4 in 8.7% (Zheng et al.).
- Self-preference. Models score their own outputs higher than others' when humans rate them equal, so do not let the model under test grade itself.
- Plain error. Used directly as a safety grader for agent runs, GPT-4o was right 75.5% of the time; a fine-tuned grader reached 91.5% (Agent-SafetyBench, 2024).
Mitigations that work: run pairwise judgments in both orders and count a win only when both agree (Zheng et al.), give the judge a reference answer, grade each dimension with a separate judge, let it answer "Unknown", and check a sample against human experts until the gap is small (Anthropic). These numbers come from 2023 and 2024 models; the method of measuring your judge still applies.
A step-by-step LLM evaluation plan
- Name the decision. Release gate, model switch or prompt change. The decision sets the pass bar.
- Write down "correct" with product and domain owners, per behavior.
- Collect 20 to 50 real cases, both directions, plus a held-out set.
- Choose graders: code first, model graders where needed, humans for calibration and high-stakes cases.
- Run each case several times and report pass rates, cost and latency, not one green run.
- Read the transcripts of failures. Sometimes the grader is wrong: Anthropic's Opus 4.5 scored 42% on CORE-Bench until the team fixed grading bugs and loosened the scaffold, then 95% (Anthropic).
- Wire it into CI and block releases on regressions.
- Sample production traffic weekly and add every new failure to the dataset.
Which LLM evaluation tools do teams use?
Usage is spread thin. Among developers who used a tool for agent observability, prompts or evals in the past year, no tool reached a third of users (Stack Overflow Developer Survey 2026). Sentry, a general monitoring tool, leads, and dedicated eval libraries such as DeepEval, Promptfoo and Ragas each sit near 5% (Stack Overflow).
Tools developers used for AI agent observability, prompts or evals
- Sentry
- 26%
- LangSmith
- 17.3%
- MLflow
- 15.2%
- Langfuse
- 15.2%
- Datadog LLM
- 11.9%
- Weights & Biases
- 10.7%
- Arize AI
- 5.8%
- Braintrust
- 5.8%
- DeepEval
- 5.8%
- Promptfoo
- 5.3%
The survey measures use, not quality. Choose a tool that fits your stack, keeps datasets portable and reports cost per run, then put your effort into the cases and graders. Anthropic's own advice is the same: frameworks are only as good as the eval tasks you run through them.
What does LLM evaluation cost?
We have not found a verified price range for LLM evaluation services, so here are the drivers:
- Dataset work: how many cases need domain experts to write or review answers.
- Run cost: every case calls the model, often several times, and model graders add calls. For scale, one public agent benchmark reports about $0.086 per task for the agent and $0.059 for the simulated user, or roughly $40 for one pass over all its tasks (tau2-bench).
- Judge design: better prompts cost more. Few-shot examples raised GPT-4's consistency from 65.0% to 77.5% but made each call 4 times more expensive (Zheng et al.).
- Human review: calibration samples and high-risk cases.
- Pipeline: a team with CI and test automation starts faster.
Common LLM evaluation mistakes
- Choosing a model from a leaderboard and never testing it on your own cases.
- An uncalibrated judge. It needs its own tests, and its order and length biases need controls.
- One run as proof. Models vary between runs; report pass rates.
- Tuning on the whole dataset. Without a held-out set you measure memory, not quality.
- No production loop. Real failures that never reach the dataset will come back.
How SDCLabs can help
We founded the open-source testing frameworks Codeception and CodeceptJS, and we apply the same discipline to AI. Our AI testing services build evaluation datasets, calibrated graders, prompt and model regression suites and red-team runs, wired into your CI. Comparing models, prompts or agent setups? AI agent benchmarking measures success rate, cost and reliability on your real tasks. Not sure where AI fits yet? Start with an AI readiness assessment.
For the failure modes these evals catch, read can you trust what your AI product says? Engagements start in 1 to 2 weeks. Tell us what you need to evaluate.
Frequently asked questions
What is LLM evaluation?
It is testing an application built on a large language model against a fixed set of real cases, scoring each answer with code checks, model graders or people, and comparing scores on every prompt, model or retrieval change.
What is the best tool for LLM evaluation?
No neutral quality ranking exists. In the Stack Overflow Developer Survey 2026, no observability, prompt or eval tool reached a third of users, so pick one that fits your stack, keeps datasets portable and reports cost per run.
Which LLM evaluation platforms track costs?
Compare platforms on whether they record tokens, latency and cost per task next to pass rates, run after run. Tracking those on a fixed set of tasks is what Anthropic recommends for baselines and regression tests.
What is LMArena LLM evaluation?
Arena, formerly LMArena and before that Chatbot Arena, ranks models from crowdsourced pairwise human votes (Arena). It is useful for a shortlist, but it does not test your prompts, data or rules.
Can you trust LLM-as-a-judge?
Only after calibration. In a 2023 study GPT-4 matched human experts on 85% of non-tie votes but kept the same verdict after answers were swapped in only 65.0% of cases (Zheng et al.), so judge in both orders and check samples against people.
Mikhail BodnarchukFounder of Codeception and CodeceptJSBoth are open-source testing frameworks: Codeception for PHP, CodeceptJS for JavaScript.LinkedInGitHub