Skip to content

Can you trust what your AI product says? Testing AI in 2026

Mykhailo PoliarushUpdated 5 min read

In 2026, testing an AI-powered product means collecting evidence that it behaves well on real inputs, because models still state false things with confidence. On one independent benchmark, hallucination rates across 26 leading models ranged from 22% to 94%, according to the Stanford AI Index 2026. Most teams that build AI agents now run some evaluations, but many still ship without them.

How often do AI products get things wrong?

Often enough that you have to measure it for your own use case. The AI Index 2026 reports two hallucination benchmarks with very different results. On AA-Omniscience, which asks 6,000 knowledge questions and lets models decline to answer, hallucination rates ranged from 22% (Grok 4.20 Beta 0305) to 94% (gpt-oss-20B), as reported in the AI Index 2026. On Vectara's summarization leaderboard, the top 15 models added false information to 1.8% to 5.4% of summaries, according to the same report. The two scales are not comparable, and that is the point: the error rate depends on the task.

Hallucination rate on AA-Omniscience: lowest and highest of 26 models

Grok 4.20 Beta 0305
22%
Claude 4.5 Haiku
26%
MiMo-V2-Pro
30%
Gemini 3 Flash
92%
gpt-oss-20B (high)
94%

Source: Stanford HAI, AI Index Report 2026, p.136 (AA-Omniscience, 26 models, April 2026)

Models also bend to the user. In a benchmark that separates knowledge from belief, GPT-4o's accuracy dropped from 98.2% to 64.4% when a false statement was presented as something the user believed, and DeepSeek R1 fell from over 90% to 14.4%, according to the AI Index 2026. If your product answers customers, a confident wrong premise in the question can produce a confident wrong answer.

What happens when AI failures reach real users?

They become incidents, refunds and court records. The AI Incident Database recorded 362 AI incidents in 2025, up from 233 in 2024, as reported in the AI Index 2026. These counts are curated by editors and lag events, so they undercount.

Courts keep their own record. The AI Hallucination Cases database maintained by Damien Charlotin listed 2,306 legal decisions where a court or tribunal found that a party relied on hallucinated content, as of its 16 November 2026 update. The OWASP Top 10 for LLM Applications uses Air Canada as its example of factual inaccuracy: the airline's chatbot gave travelers wrong information, and "the airline was successfully sued as a result."

Engineers see the same pattern in their own tools. In the Stack Overflow Developer Survey 2025, 66% of developers named "AI solutions that are almost right, but not quite" as a frustration. Almost right is the hardest kind of wrong to catch without tests.

What are the main challenges in testing AI products?

Classic test automation assumes one input gives one correct output. AI products break that assumption in several ways.

  • Quality is hard to define. In LangChain's State of Agent Engineering survey of 1,340 practitioners, 32% named quality as the top barrier to production. LangChain defines quality as accuracy, relevance, consistency, tone and policy adherence, which means five things to test, not one.
  • Outputs vary between runs. The same prompt can pass today and fail tomorrow, so one green run proves little. Tests need repeated runs and pass rates, not single assertions.
  • Inputs are hostile. OWASP ranks prompt injection as risk number one and notes that RAG and fine-tuning "do not fully mitigate" it. Injected instructions can hide in a web page, a PDF or an email that the model reads.
  • Teams lack the know-how. The main obstacles to implementing responsible AI are gaps in knowledge (59%), budget constraints (48%) and regulatory uncertainty (41%), according to the AI Index 2026. The same report notes that reporting on responsible AI benchmarks "remains sparse" even among frontier model developers.

Which security risks should an AI test plan cover?

The OWASP Top 10 for LLM Applications 2025, released on 18 November 2024, is the most practical checklist available. It covers the model and everything around it: the data you feed it, the tools you let it call and the output you pass downstream.

OWASP Top 10 for LLM Applications 2025

  • LLM01 Prompt injection
  • LLM02 Sensitive information disclosure
  • LLM03 Supply chain
  • LLM04 Data and model poisoning
  • LLM05 Improper output handling
  • LLM06 Excessive agency
  • LLM07 System prompt leakage
  • LLM08 Vector and embedding weaknesses
  • LLM09 Misinformation
  • LLM10 Unbounded consumption

Three items deserve special attention in agent products. Excessive agency (LLM06) is about what the model is allowed to do, so test permissions, not just answers. Improper output handling (LLM05) is about what your code does with model output, so treat it like untrusted user input. Unbounded consumption (LLM10) is about cost and denial of service, so test limits on tokens, retries and tool loops.

How are teams testing AI today?

Teams combine people and models. The LangChain survey reports human review at 59.8% and LLM-as-judge scoring at 53.3%, and 52.4% of respondents run offline evaluations on test sets. The share of respondents not evaluating at all was 29.5% overall and 22.8% among teams that already have agents in production. LangChain sells evaluation tooling and surveyed its own audience, so treat these as figures from a vendor and from a group that is ahead of the average.

Standards help structure the work. The NIST Generative AI Profile (AI 600-1), published on 26 July 2024, maps generative AI risks to the AI Risk Management Framework. It is voluntary, but it gives product, security and compliance teams a shared vocabulary.

What does good AI testing look like?

Good AI testing looks like good software testing, adapted to probabilistic output. A practical starting point:

  1. Build a golden dataset from real use. Collect real questions, documents and edge cases, write down what a good answer looks like, and version the set like code.
  2. Test the system, not only the model. Retrieval, prompts, tools, permissions and post-processing all fail on their own. A better model will not fix a broken retriever.
  3. Run each case several times. Report pass rates and track variance, so a lucky run does not hide a flaky behavior.
  4. Calibrate LLM-as-judge against people. Use model graders for scale, check a sample by hand, and keep human review for high-stakes answers.
  5. Red-team with the OWASP list. Write test cases for prompt injection, data leakage, excessive agency and runaway cost before launch, then keep them in the regression suite.
  6. Gate releases and watch production. Run the eval suite in CI on every prompt or model change, and sample production traffic for online evaluation.

Want a second opinion on your AI product?

If your team is shipping an assistant, a RAG search or an agent and cannot say how often it is wrong, that is the problem SDCLabs' AI testing services are for. We founded the open-source testing frameworks Codeception and CodeceptJS, and our senior engineers run AI-assisted workflows in which automated tests verify every change. Bring us your use case and we will talk through eval sets, OWASP risks and how to put the checks into your pipeline. Engagements start in 1 to 2 weeks. Tell us what you are building.

Comparing agents or models? See AI agent benchmarking and our state of AI engineering in 2026 report.

Sources checked on 2026-11-17.

Frequently asked questions

How often do AI models hallucinate in 2026?

It depends on the task. On AA-Omniscience, hallucination rates across 26 models ranged from 22% to 94%, while the top 15 models on Vectara's summarization leaderboard ranged from 1.8% to 5.4%, according to the AI Index 2026.

What is the OWASP Top 10 for LLM Applications 2025?

It is OWASP's list of the ten main security risks for LLM applications, released on 18 November 2024, with prompt injection as LLM01. See the OWASP list.

How do teams evaluate AI agents today?

In LangChain's survey of 1,340 practitioners, 52.4% run offline evaluations on test sets, and human review (59.8%) and LLM-as-judge scoring (53.3%) are both common. 29.5% do not evaluate at all.

Where should an AI testing effort start?

Start with a golden dataset built from real use, test the whole system rather than only the model, run each case several times, red-team the OWASP risks and run the eval suite on every prompt or model change.

Related posts

Get an estimate for your product.