Skip to content

What is agent observability?

Sergey Almyashev

Definition

Agent observability is the ability to reconstruct what an AI agent did on each run: every model call, tool call, handoff and decision, with inputs, outputs, tokens, cost and latency, recorded as linked traces. It lets a team debug a failed task, notice when behavior drifts and turn real runs into new test cases.

How is agent observability different from LLM monitoring?

LLM monitoring watches single requests. An agent run is a tree of them: the model plans, calls a tool, reads the result, calls the model again, maybe hands off to another agent. Agent observability records that tree so you can follow one task from start to finish.

LLM request monitoringAgent observability
Unit of recordOne model callOne run: every step and how the steps connect
Typical questionIs the endpoint slow or failing?Why did this run pick the wrong tool on step four?
Key signalsLatency, error rate, tokens per callSteps per run, tool calls and results, handoffs, cost per completed task
Main useUptime and spend alertsDebugging, finding drift, turning failed runs into test cases

What should an agent trace contain?

The vendor-neutral answer is the OpenTelemetry GenAI semantic conventions. They define span names and attributes so traces from different frameworks and backends line up. The agent-related operations:

Operation (gen_ai.operation.name)What the span represents
create_agentCreating an agent, for example on a hosted agent service
invoke_agentAn agent run where the model drives control flow through tool calls
invoke_workflowA run where the application defines control flow, such as a graph of steps
planA planning or task decomposition phase, when the instrumentation can tell it apart
chat and other inference operationsOne model call, named chat plus the model, with token usage (client inference)
execute_toolOne tool call, named execute_tool plus the tool name (execute tool)

The split between invoke_agent and invoke_workflow mirrors the difference between an agent and an agentic workflow: who decides the next step.

Two facts to know before you build on these conventions:

  • They are still marked Development, not stable, so attribute names can change.
  • They moved. In semantic conventions v1.42.0, released June 12, 2026, all gen_ai.* definitions were deprecated in the core repository and moved to the dedicated GenAI repository. Pages on opentelemetry.io now point there.

The conventions also take a position on privacy. Model instructions, user messages and outputs are treated as sensitive, and instrumentations should not capture them by default but offer an opt-in (GenAI spans). Tool call arguments and results are opt-in attributes for the same reason.

Example: one agent run as OpenTelemetry spans

This Python script records a run with the OpenTelemetry SDK: a parent invoke_agent span, a child model call with token counts, and a child tool call. We ran it with opentelemetry-sdk 1.45.2 and the console exporter; in production you would swap in an OTLP exporter pointed at your tracing backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import SpanKind

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

# One agent run: the agent span is the parent, model and tool calls are its children.
with tracer.start_as_current_span("invoke_agent support_agent", kind=SpanKind.INTERNAL) as run:
    run.set_attribute("gen_ai.operation.name", "invoke_agent")
    run.set_attribute("gen_ai.agent.name", "support_agent")
    run.set_attribute("gen_ai.conversation.id", "conv-7731")

    with tracer.start_as_current_span("chat claude-sonnet-5-5", kind=SpanKind.CLIENT) as llm:
        llm.set_attribute("gen_ai.operation.name", "chat")
        llm.set_attribute("gen_ai.provider.name", "anthropic")
        llm.set_attribute("gen_ai.request.model", "claude-sonnet-5-5")
        llm.set_attribute("gen_ai.usage.input_tokens", 812)
        llm.set_attribute("gen_ai.usage.output_tokens", 64)

    with tracer.start_as_current_span("execute_tool search_orders", kind=SpanKind.INTERNAL) as tool:
        tool.set_attribute("gen_ai.operation.name", "execute_tool")
        tool.set_attribute("gen_ai.tool.name", "search_orders")
        tool.set_attribute("gen_ai.tool.call.id", "call_01")
        # Arguments and results are opt-in: they may contain personal data.

The exporter printed three spans. The chat and execute_tool spans carry the invoke_agent span's ID as their parent, so a tracing UI shows one run with two steps. The token counts here are illustrative; in a real agent, framework instrumentation fills them from the provider's response.

How common is agent tracing?

Common among teams that already ship agents, by one vendor's count. LangChain's 2026 agent engineering survey collected 1,340 responses between November 18 and December 2, 2025. In it, 89% of respondents had some observability for their agents and 62% had detailed tracing of individual steps and tool calls. Among respondents with agents in production, those figures rose to 94% and 71.5%. LangChain sells tracing tools and ran the survey itself, so read these as a signal from agent builders, not a market average.

What to check before you rely on agent traces

  1. Every step is a span, including retries, guardrail blocks and handoffs, not only model calls.
  2. Content capture is a decision, not a default. Decide what you record, mask personal data, and set retention.
  3. Cost and outcome sit on the same trace, so you can see cost per completed task, not only cost per call.
  4. Failed runs feed the test suite. A trace that shows a wrong turn is a ready-made case for your AI evals.

For where tracing sits among evals, MCP and inference costs, see our 2026 AI engineering report. On the AI agents we build at SDCLabs, traces sit next to the automated tests: the tests verify every change, engineers decide what ships, and the traces explain what happened when a run fails.

Sources

  1. Semantic conventions for GenAI agent and framework spans, OpenTelemetry (semantic-conventions-genai repository), 2026-11
  2. Semantic conventions for GenAI client inference, OpenTelemetry, 2026-11
  3. Execute tool, OpenTelemetry, 2026-11
  4. Semantic conventions for generative client AI spans, OpenTelemetry, 2026-11
  5. Semantic Conventions v1.42.0 release, OpenTelemetry (GitHub), 2026-06-12
  6. State of Agent Engineering, LangChain (vendor survey), 2026-06-12

Frequently asked questions

What is the difference between agent observability and LLM observability?

LLM observability usually tracks individual model calls: latency, errors and tokens. Agent observability links those calls with tool calls, handoffs and plans into one trace per task, so you can see why a multi-step run went wrong.

Is OpenTelemetry ready for AI agents?

Partly. The GenAI semantic conventions define agent, workflow, planning, model and tool spans, but they are still marked Development, so attribute names can change. Pin the convention version your instrumentation uses.

Should traces store prompts and model outputs?

Only by deliberate choice. The OpenTelemetry conventions treat them as sensitive and off by default. If you need them for debugging, capture a sample of runs, mask personal data and set a short retention period.

How does agent observability help with evals?

Traces show where real runs fail. Each failed run can become a new eval task with the same input and a clear success criterion, so the next release is tested on the failure that actually happened.

Sergey AlmyashevSergey AlmyashevChief Operating Officer14+ years in QA, automated, exploratory and performance testing, and continuous integration and delivery within Scrum, helping teams in product and outsourcing companies organize teams, processes and tools.LinkedIn

Building with AI agents? Talk to an engineer.