Skip to content

How to build AI agents that still work after the demo

Mykhailo Poliarush7 min read

How to build AI agents that work in production comes down to a few engineering choices: use the simplest design that can do the job, give one capable model a small set of well-designed tools and clear instructions, then add memory, guardrails, evals and human approval before you add autonomy. The model is rarely the hard part. Tools, tests and supervision are.

Do you need an agent at all?

Often not. Anthropic, after working with dozens of teams, recommends finding the simplest solution possible, which "might mean not building agentic systems at all", because agentic systems trade latency and cost for better task performance (Anthropic, Building effective agents). A fixed workflow with one or two model calls is easier to test and cheaper to run. We explain the difference between workflows and agents in what is AI agent development?

OpenAI's guide gives three signs that an agent is worth it: decisions that need judgment and exceptions, rule sets that have become too hard to maintain, and heavy reliance on unstructured data such as documents and conversations. "Otherwise, a deterministic solution may suffice" (OpenAI, A practical guide to building agents). The OWASP agentic security list makes the same point from the attacker's side: avoid unnecessary autonomy, because it widens the attack surface without adding value (OWASP, December 2025).

What is the architecture of an AI agent?

Strip away the frameworks and an agent has three parts: a model, tools and instructions (OpenAI). It runs in a loop: the model reads the goal and the context, calls a tool, reads the result, and decides the next step. Anthropic describes agents as "typically just LLMs using tools based on environmental feedback in a loop" (Anthropic).

  • Model. Prototype with the most capable model to set a baseline with evals, then try smaller models where they still meet the bar (OpenAI).
  • Instructions. Turn existing procedures, support scripts and policies into numbered steps, with a branch for each common edge case.
  • Tools. Data tools that read (databases, documents, search), action tools that change things (update a record, send a message), and other agents used as tools.
  • Exit conditions. A final answer, a structured output, an error or a maximum number of turns. Without a cap, a confused agent keeps spending.

How should you design an agent's tools?

Treat tools as the agent's user interface. While building its coding agent for SWE-bench, Anthropic "spent more time optimizing our tools than the overall prompt"; one fix, requiring absolute file paths, removed a whole class of mistakes (Anthropic).

Rules from Anthropic's later guide on writing tools for agents:

  • Fewer, task-shaped tools. More tools do not always lead to better outcomes. A search_contacts tool beats a list_contacts tool that dumps every record into the context.
  • Clear names and boundaries. Namespace related tools by service and resource so the agent picks the right one.
  • Meaningful outputs. Return names instead of opaque IDs and cap response size; Claude Code limits tool responses to 25,000 tokens by default.
  • Watch the errors. Many invalid-parameter errors mean the description is unclear.

How many tools is too many depends on overlap. OpenAI has seen agents handle more than 15 distinct tools and others struggle with fewer than 10 overlapping ones (OpenAI). A widely used standard for connecting tools is the Model Context Protocol; Anthropic donated it to the Linux Foundation's Agentic AI Foundation in December 2025.

How do agents handle memory and context?

The context window is a budget, not a bin. Anthropic calls the effect "context rot": as the number of tokens grows, the model recalls information from it less accurately (Anthropic, Effective context engineering). Three techniques help long tasks:

  • Compaction: summarize the conversation as it nears the limit and continue in a fresh window.
  • Structured notes (agentic memory): the agent writes progress and decisions to storage outside the context and reads them back.
  • Just-in-time retrieval: keep file paths, queries or links, and load the content only when it is needed.

Treat stored memory as untrusted input. OWASP lists memory and context poisoning as a top agentic risk (OWASP).

Single agent or multi-agent?

Start with one. OpenAI's general recommendation is to maximize a single agent's capabilities first, and split only when instructions get too complex or the agent keeps choosing the wrong tools (OpenAI). When you do split, OpenAI sees two broadly applicable patterns: a manager agent that calls specialists as tools, or peers that hand off a conversation to each other.

Multi-agent systems cost more to run. In Anthropic's own measurements, agents use about 4 times more tokens than chat, and multi-agent systems about 15 times more (Anthropic). We break down what that means for a product in AI agents in products. Splitting agents by job role (planner, developer, tester) adds handoffs that lose context; we compare that design with one agent that loads skills in AI agent architecture: skills and a loop, not role agents.

Which framework should you build an agent with?

You can build without one. Anthropic suggests starting with model APIs directly, because frameworks add layers that can hide the prompts and responses you need to debug (Anthropic). Among developers who used an agent framework in the past year, LangChain leads at 30.2%, ahead of the OpenAI Agents SDK at 23.8%, and 18.9% report "Terminal Use", which Stack Overflow reads as a preference for direct, lightweight orchestration (Stack Overflow Developer Survey 2026).

Agent orchestration tools and frameworks developers used in the past year

LangChain
30.2%
OpenAI Agents SDK
23.8%
Terminal Use
18.9%
LangGraph
18.8%
Llama Stack
17.4%
Google ADK
14%
Hermes Agent
10.2%
Vercel AI SDK
9%
Pydantic AI
8.9%
CrewAI
5.5%

Source: Stack Overflow Developer Survey 2026, AI, agent orchestration tools (n=4,400)

Pick the framework your team can read and debug. Keep tools and evals independent of it, so you can switch later.

Which guardrails does an AI agent need?

Layer them; no single check is enough. OpenAI's guide lists seven types, from relevance and safety classifiers to tool safeguards, and pairs them with normal authentication, authorization and access control (OpenAI).

Seven types of agent guardrails

Seven types of agent guardrails
CriterionGuardrailWhat it does
Relevance classifierFlags off-topic requests so the agent stays in scope
Safety classifierDetects jailbreaks and prompt injections
PII filterChecks output for unnecessary personal data
ModerationFlags harmful or inappropriate inputs
Tool safeguardsRates each tool low, medium or high risk and pauses or escalates risky calls
Rules-based protectionsBlocklists, input length limits and regex filters for known threats
Output validationChecks responses against brand and content rules

Source: OpenAI, A practical guide to building agents (April 2025; model vendor)

The most important guardrail is permissions. OWASP's agentic list, built with more than 100 experts, lists agent goal hijack first and tool misuse second, and recommends a least-privilege profile for every tool: read-only database queries, no send or delete rights for an email summarizer (OWASP). Rate each tool low, medium or high risk, and require a human for high-risk or irreversible actions such as refunds, cancellations and payments. Many teams already work this way: 60% of respondents to a Stack Overflow pulse survey block agents from unapproved system changes (Stack Overflow, May 2026).

How do you evaluate and supervise an AI agent?

Build the evals before the agent, and keep people in the loop after launch. OpenAI's first rule for choosing a model is to set up evals to establish a baseline (OpenAI). Quality is the top barrier to production for 32% of agent builders in LangChain's vendor survey (LangChain), and only tests show whether a change helped. We cover the test plan in AI agent testing.

Supervision is a design feature, not a fallback. OpenAI names two triggers for handing a task to a person: the agent exceeds a failure threshold, such as several attempts to understand the customer, or it reaches a sensitive, irreversible or high-stakes action. Trace every run, review samples weekly and widen autonomy only when the numbers support it. Most technologists already hold back: 63% rarely or never let agents run entirely on autopilot (Stack Overflow, May 2026).

How to build AI agents step by step

  1. Pick one workflow with a clear owner, a measurable outcome and real volume.
  2. Write the success criteria and 20 to 50 test tasks from real cases before writing code (see LLM evaluation).
  3. Build the simplest version: one model, a few tools, numbered instructions, a turn limit.
  4. Design and test the tools until invalid calls are rare.
  5. Add memory only where tasks outlive the context window.
  6. Add guardrails and approvals: least-privilege tools, input and output checks, a human on high-risk actions.
  7. Test the agent end to end in a sandbox, several runs per task, with cost and latency tracked (see AI agent testing).
  8. Ship to a small group with tracing on, read transcripts weekly and turn every failure into a new test.

What drives the cost of building an AI agent?

We do not publish price ranges we cannot verify. The drivers are integration work (APIs, permissions, legacy systems), tool design and testing, the eval suite, model and token costs per task, supervision and review time, and the risk level of the actions the agent can take. Cost control matters: Gartner forecasts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls (Gartner, June 2025).

Common mistakes when building AI agents

  • Starting multi-agent. Complexity before evidence that one agent is not enough.
  • Wrapping every API endpoint as a tool. Agents need task-shaped tools.
  • Broad credentials. An agent with admin rights turns one prompt injection into an incident.
  • No evals until launch. Without a baseline, every change is a guess.
  • No human path. Agents need a way to hand a task back when they are stuck.

How SDCLabs can help

Our senior engineers build agents with AI-assisted workflows and keep every agent supervised, tested and accountable. Custom AI agent development builds agents into your product, with guardrails and human approvals. Our AI agent development services cover the integrations, cloud and data work around them. Before launch, AI agent benchmarking measures success rate, cost and reliability on your real tasks. If you are still choosing a first use case, start with an AI readiness assessment.

Engagements start in 1 to 2 weeks. Tell us which workflow you want to automate.

Frequently asked questions

What is an AI agent in software development?

A system in which a model directs its own steps and tool use in a loop to complete a task, as opposed to a workflow that follows predefined code paths (Anthropic).

What platforms support AI agent development?

You can build directly on model APIs or use a framework. Among developers who used one in the past year, LangChain (30.2%) and the OpenAI Agents SDK (23.8%) lead (Stack Overflow Developer Survey 2026). Pick one your team can debug.

Do you need a framework to build an AI agent?

No. Anthropic suggests starting with model APIs directly, because frameworks can hide the prompts and responses you need to debug (Anthropic). Keep tools and evals independent of whatever you choose.

Should you build a single agent or a multi-agent system?

Start with one agent and split only when instructions or tool choice break down (OpenAI). Multi-agent systems use about 15 times more tokens than chat in Anthropic's measurements (Anthropic).

What guardrails does an AI agent need?

Layered checks on inputs and outputs, least-privilege permissions for every tool, a risk rating per tool and a human approval step for high-risk or irreversible actions, as recommended by OWASP and OpenAI.

#ai-agents#agentic-engineering#ai-agent-development
Mykhailo PoliarushMykhailo PoliarushFounder and CEOLeads SDCLabs, an AI development company founded in Krakow in 2011.LinkedInGitHub

Related posts

Get an estimate for your product.