# What is agent observability? Traces, spans and an example

URL: https://sdclabs.com/glossary/agent-observability/

> Agent observability records each model call, tool call and handoff of an agent run as linked traces. What a trace holds, OpenTelemetry spans, an example.

# What is agent observability?

[Sergey Almyashev](https://sdclabs.com/author/almyashev/) 2026-11-21

Definition

Agent observability is the ability to reconstruct what an AI agent did on each run: every model call, tool call, handoff and decision, with inputs, outputs, tokens, cost and latency, recorded as linked traces. It lets a team debug a failed task, notice when behavior drifts and turn real runs into new test cases.

## How is agent observability different from LLM monitoring?

LLM monitoring watches single requests. An agent run is a tree of them: the model plans, calls a tool, reads the result, calls the model again, maybe hands off to another agent. Agent observability records that tree so you can follow one task from start to finish.

|   | LLM request monitoring | Agent observability |
|---|---|---|
| Unit of record | One model call | One run: every step and how the steps connect |
| Typical question | Is the endpoint slow or failing? | Why did this run pick the wrong tool on step four? |
| Key signals | Latency, error rate, tokens per call | Steps per run, tool calls and results, handoffs, cost per completed task |
| Main use | Uptime and spend alerts | Debugging, finding drift, turning failed runs into test cases |

## What should an agent trace contain?

The vendor-neutral answer is the [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) . They define span names and attributes so traces from different frameworks and backends line up. The agent-related operations:

| Operation (gen_ai.operation.name) | What the span represents |
|---|---|
| create_agent | Creating an agent, for example on a hosted agent service |
| invoke_agent | An agent run where the model drives control flow through tool calls |
| invoke_workflow | A run where the application defines control flow, such as a graph of steps |
| plan | A planning or task decomposition phase, when the instrumentation can tell it apart |
| chat and other inference operations | One model call, named chat plus the model, with token usage ([client inference](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md) ) |
| execute_tool | One tool call, named execute_tool plus the tool name ([execute tool](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/execute-tool.md) ) |

The split between invoke_agent and invoke_workflow mirrors the difference between an agent and an [agentic workflow](https://sdclabs.com/glossary/agentic-workflow/) : who decides the next step.

Two facts to know before you build on these conventions:

- **They are still marked Development** , not stable, so attribute names can change.
- **They moved.** In semantic conventions [v1.42.0](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.42.0) , released June 12, 2026, all gen_ai.* definitions were deprecated in the core repository and moved to the dedicated GenAI repository. Pages on opentelemetry.io now point there.

The conventions also take a position on privacy. Model instructions, user messages and outputs are treated as sensitive, and instrumentations should not capture them by default but offer an opt-in ([GenAI spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) ). Tool call arguments and results are opt-in attributes for the same reason.

## Example: one agent run as OpenTelemetry spans

This Python script records a run with the OpenTelemetry SDK: a parent invoke_agent span, a child model call with token counts, and a child tool call. We ran it with opentelemetry-sdk 1.45.2 and the console exporter; in production you would swap in an OTLP exporter pointed at your tracing backend.

from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter from opentelemetry.trace import SpanKind provider = TracerProvider() provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter())) trace.set_tracer_provider(provider) tracer = trace.get_tracer("support-agent") # One agent run: the agent span is the parent, model and tool calls are its children. with tracer.start_as_current_span("invoke_agent support_agent", kind=SpanKind.INTERNAL) as run: run.set_attribute("gen_ai.operation.name", "invoke_agent") run.set_attribute("gen_ai.agent.name", "support_agent") run.set_attribute("gen_ai.conversation.id", "conv-7731") with tracer.start_as_current_span("chat claude-sonnet-5-5", kind=SpanKind.CLIENT) as llm: llm.set_attribute("gen_ai.operation.name", "chat") llm.set_attribute("gen_ai.provider.name", "anthropic") llm.set_attribute("gen_ai.request.model", "claude-sonnet-5-5") llm.set_attribute("gen_ai.usage.input_tokens", 812) llm.set_attribute("gen_ai.usage.output_tokens", 64) with tracer.start_as_current_span("execute_tool search_orders", kind=SpanKind.INTERNAL) as tool: tool.set_attribute("gen_ai.operation.name", "execute_tool") tool.set_attribute("gen_ai.tool.name", "search_orders") tool.set_attribute("gen_ai.tool.call.id", "call_01") # Arguments and results are opt-in: they may contain personal data.
The exporter printed three spans. The chat and execute_tool spans carry the invoke_agent span's ID as their parent, so a tracing UI shows one run with two steps. The token counts here are illustrative; in a real agent, framework instrumentation fills them from the provider's response.

## How common is agent tracing?

Common among teams that already ship agents, by one vendor's count. LangChain's [2026 agent engineering survey](https://www.langchain.com/state-of-agent-engineering) collected 1,340 responses between November 18 and December 2, 2025. In it, 89% of respondents had some observability for their agents and 62% had detailed tracing of individual steps and tool calls. Among respondents with agents in production, those figures rose to 94% and 71.5%. LangChain sells tracing tools and ran the survey itself, so read these as a signal from agent builders, not a market average.

## What to check before you rely on agent traces

1. **Every step is a span** , including retries, guardrail blocks and handoffs, not only model calls.
1. **Content capture is a decision** , not a default. Decide what you record, mask personal data, and set retention.
1. **Cost and outcome sit on the same trace** , so you can see cost per completed task, not only cost per call.
1. **Failed runs feed the test suite.** A trace that shows a wrong turn is a ready-made case for your [AI evals](https://sdclabs.com/glossary/ai-evals/) .

For where tracing sits among evals, MCP and inference costs, see our [2026 AI engineering report](https://sdclabs.com/blog/state-of-ai-engineering-2026/) . On the [AI agents](https://sdclabs.com/services/ai-agents/) we build at SDCLabs, traces sit next to the automated tests: the tests verify every change, engineers decide what ships, and the traces explain what happened when a run fails.

## Sources

1. [Semantic conventions for GenAI agent and framework spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) , OpenTelemetry (semantic-conventions-genai repository), 2026-11
1. [Semantic conventions for GenAI client inference](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/client-inference.md) , OpenTelemetry, 2026-11
1. [Execute tool](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/execute-tool.md) , OpenTelemetry, 2026-11
1. [Semantic conventions for generative client AI spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) , OpenTelemetry, 2026-11
1. [Semantic Conventions v1.42.0 release](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.42.0) , OpenTelemetry (GitHub), 2026-06-12
1. [State of Agent Engineering](https://www.langchain.com/state-of-agent-engineering) , LangChain (vendor survey), 2026-06-12

## Frequently asked questions

### What is the difference between agent observability and LLM observability?

LLM observability usually tracks individual model calls: latency, errors and tokens. Agent observability links those calls with tool calls, handoffs and plans into one trace per task, so you can see why a multi-step run went wrong.

### Is OpenTelemetry ready for AI agents?

Partly. The GenAI semantic conventions define agent, workflow, planning, model and tool spans, but they are still marked Development, so attribute names can change. Pin the convention version your instrumentation uses.

### Should traces store prompts and model outputs?

Only by deliberate choice. The OpenTelemetry conventions treat them as sensitive and off by default. If you need them for debugging, capture a sample of runs, mask personal data and set a short retention period.

### How does agent observability help with evals?

Traces show where real runs fail. Each failed run can become a new eval task with the same input and a clear success criterion, so the next release is tested on the failure that actually happened.

Services

- [AI agents](https://sdclabs.com/services/ai-agents/)
- [DevOps and TestOps](https://sdclabs.com/services/devops-and-testops/)

Read next

- [The state of AI engineering in 2026](https://sdclabs.com/blog/state-of-ai-engineering-2026/)
- [AI agents can take the first page, but humans still own the outage](https://sdclabs.com/blog/ai-agents-it-operations/)

Related terms

- [AI evals](https://sdclabs.com/glossary/ai-evals/)
- [LLMOps](https://sdclabs.com/glossary/llmops/)
- [AI guardrails](https://sdclabs.com/glossary/ai-guardrails/)
- [Agentic workflow](https://sdclabs.com/glossary/agentic-workflow/)

[Sergey Almyashev](https://sdclabs.com/author/almyashev/) Chief Operating Officer14+ years in QA, automated, exploratory and performance testing, and continuous integration and delivery within Scrum, helping teams in product and outsourcing companies organize teams, processes and tools.[LinkedIn](https://www.linkedin.com/in/sergeyalmyashev/)

## Building with AI agents? Talk to an engineer.

[Start a project](https://sdclabs.com/contact/) [Book a call with the CEO](https://calendly.com/sdclabs/60min)
