# What is agentic RAG? How it works, with a runnable example

URL: https://sdclabs.com/glossary/agentic-rag/

> Agentic RAG lets an agent decide when to retrieve, grade results and rewrite queries. How it differs from classic RAG, with code and production checks.

# What is agentic RAG?

[Mykhailo Poliarush](https://sdclabs.com/author/poliarush/) 2026-11-19

Definition

Agentic RAG is retrieval-augmented generation run by an AI agent instead of a fixed pipeline. The agent decides whether to search, chooses sources and queries, checks whether the retrieved text is relevant, and rewrites or splits the query when it is not, before writing a grounded answer. It trades extra latency and tokens for better answers on complex questions.

## How does agentic RAG work?

Classic retrieval-augmented generation runs one fixed pass: embed the question, fetch the top chunks, write the answer. Agentic RAG puts a model in charge of that pass. The model decides whether to search at all, judges what came back, and searches again with a better query when the first results miss.

The [LangGraph RAG agent tutorial](https://docs.langchain.com/oss/python/langgraph/agentic-rag) builds the smallest useful version as a graph of four nodes and one routing function:

| Step | Node or function in the tutorial | What it decides |
|---|---|---|
| 1. Route | generate_query_or_respond | Answer directly, or call the retriever tool. The tutorial calls this node "the agent decision point". |
| 2. Retrieve | retrieve | Runs the retriever tool the model asked for. |
| 3. Grade | grade_documents (routing function) | A grader model returns a binary yes or no: is the document relevant to the question? |
| 4. Rewrite | rewrite_question | On "no", reformulates the question and loops back to step 1. |
| 5. Answer | generate_answer | On "yes", answers from the retrieved context, three sentences at most. |

Larger systems add more moves on top of this skeleton. A [survey of agentic RAG](https://arxiv.org/abs/2501.09136) (Singh et al., first posted January 2025, revised April 2026) names four design patterns that agents bring into the pipeline: reflection, planning, tool use and multi-agent collaboration. It classifies architectures by four axes: how many agents there are, the control structure, the degree of autonomy, and how knowledge is represented. The loop that drives each agent is the same think, call a tool, read the result cycle described on our [ReAct agent](https://sdclabs.com/glossary/react-agent/) page.

## How is agentic RAG different from traditional RAG?

| Question | Traditional RAG | Self-RAG (2023) | Agentic RAG |
|---|---|---|---|
| Who decides to retrieve? | The pipeline, every time | The model, through trained reflection tokens | An agent, per step |
| How many queries per question? | One | Zero or more, on demand | Many: rewrites, subqueries, other tools |
| Are results checked? | No | The model critiques passages and its own output | A grader or the agent itself checks them |
| What happens on a bad result? | The answer uses it anyway | The model's self-critique flags it | Rewrite, try another source, or stop |
| Needs a custom model? | No | Yes, a fine-tuned model | No, any model with tool calling |

[Self-RAG](https://arxiv.org/abs/2310.11511) (Asai et al., October 2023) is a useful midpoint. It trains a single model to retrieve passages on demand and to critique them with special reflection tokens, so it is adaptive without being an agent loop. Its abstract makes the case against fixed retrieval: pulling in a fixed number of passages whether or not they are needed or relevant can make the answer worse. The agent approach reaches the same adaptivity without retraining, by putting the checks in the orchestration code.

## Example: a retrieval agent loop you can run offline

The routing, grading and rewriting steps are stubs here, so the script runs with no API key or vector store. Swap each stub for a model call to get the LangGraph design above.

"""Agentic RAG control loop: decide, retrieve, grade, then rewrite or answer. The LLM steps are stubs so this runs offline; replace them with model calls.""" import re DOCS = [ "Traditional RAG runs one static retrieve-then-generate pass per question.", "Self-RAG trains a model to emit reflection tokens; posted to arXiv in October 2023.", "Azure agentic retrieval splits a complex query into subqueries that run in parallel.", ] def needs_retrieval(question): # stub for the model's routing decision return not question.lower().startswith(("hi", "thanks")) def words(text): return set(re.findall(r"[a-z0-9-]+", text.lower())) def retrieve(query, k=1): # toy keyword retriever return sorted(DOCS, key=lambda d: -len(words(query) & words(d)))[:k] def grade(question, doc): # stub grader: the doc must mention the asked-about idea return "reflection" in question.lower() and "reflection" in doc.lower() def rewrite(query): # stub for an LLM query rewrite return query + " self-rag reflection tokens" def answer(question, max_rewrites=2): if not needs_retrieval(question): return "direct answer, no retrieval", [] query, trace = question, [] for attempt in range(max_rewrites + 1): doc = retrieve(query)[0] ok = grade(question, doc) trace.append((attempt, query, doc, ok)) if ok: return f"From the source: {doc}", trace query = rewrite(query) return "no grounded answer found, say so instead of guessing", trace if __name__ == "__main__": result, trace = answer("When was the RAG paper with reflection published?") for attempt, query, doc, ok in trace: print(f"try {attempt}: query={query!r}\n doc={doc!r}\n relevant={ok}") print(result) print(answer("hi there")[0])
Output with Python 3:

try 0: query='When was the RAG paper with reflection published?' doc='Traditional RAG runs one static retrieve-then-generate pass per question.' relevant=False try 1: query='When was the RAG paper with reflection published? self-rag reflection tokens' doc='Self-RAG trains a model to emit reflection tokens; posted to arXiv in October 2023.' relevant=True From the source: Self-RAG trains a model to emit reflection tokens; posted to arXiv in October 2023. direct answer, no retrieval
The first search is a tie: one document shares the word "RAG", another shares "reflection", and the toy retriever keeps the first. A classic pipeline would answer from that wrong passage. Here the grader rejects it, the rewrite adds the missing terms, and the second search finds the right passage. The greeting skips retrieval entirely. When every rewrite fails, the function says so instead of answering from weak context.

## What does managed agentic retrieval look like?

Search platforms now ship the pattern as a service. Microsoft's [agentic retrieval in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview) (doc dated October 29, 2026) can use an LLM to plan queries at its low and medium effort levels. It splits a complex question into focused subqueries and runs them in parallel. Each subquery can be keyword, vector or hybrid search and is semantically reranked. The results are merged into one grounding set, with optional source references and an activity log.

The same page is direct about the cost side. Agentic retrieval "adds latency compared to a single-query pipeline", and billing moves from a cost per query to a variable cost per token that depends on the reasoning effort you set (minimal, low or medium in the GA REST version 2026-10-01). LLM query planning is billed separately as model tokens.

## When is agentic RAG worth it, and when is classic RAG enough?

Microsoft's examples of questions that justify the extra step: several asks in one question, questions that depend on earlier turns in the chat, queries that need rewriting or synonyms, and spelling mistakes. Add to that any question that spans more than one source, such as a policy document plus a live database.

Stay with a single retrieval pass when questions are short fact lookups against one well-indexed corpus, when you need a tight latency budget, or when you cannot afford token costs that vary per question. Fix the data first, too. Agents searching messy, unlabeled content pay for more failed searches per question; our report on [AI-ready data](https://sdclabs.com/blog/ai-ready-data/) covers what that preparation involves.

## What should you check before shipping a retrieval agent?

- **The grader is a model too.** Build a small labeled set of relevant and irrelevant chunks and measure the grader on it. A lenient grader turns the agent back into a classic pipeline; a strict one loops until the cap.
- **Caps on rewrites and tool calls** , with an honest "not found" when the cap is hit.
- **Retrieved text is data, not instructions.** The LangGraph answer prompt tells the model to treat context as data and ignore any instructions inside it, a basic guard against prompt injection through documents.
- **Citations survive every hop.** Keep source references from each subquery through to the answer.
- **Query logs.** Record which queries went to which sources. Azure exposes this as the activity log; in your own stack, write it to traces.
- **Evaluation of the whole chain.** The survey lists evaluation, coordination, memory management, efficiency and governance as open problems. Score retrieval and answers separately, as in our guide to [LLM evaluation](https://sdclabs.com/blog/llm-evaluation/) .

SDCLabs designs and tests retrieval agents as part of its [AI agents](https://sdclabs.com/services/ai-agents/) work. Senior engineers set the guardrails and review what ships, and automated tests verify each change.

## Sources

1. [Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (arXiv 2501.09136, v4)](https://arxiv.org/abs/2501.09136) , Singh, Ehtesham, Kumar, Khoei, Vasilakos, 2026-04-01
1. [Build a custom RAG agent with LangGraph (agentic RAG tutorial)](https://docs.langchain.com/oss/python/langgraph/agentic-rag) , LangChain, 2026-09-23
1. [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (arXiv 2310.11511)](https://arxiv.org/abs/2310.11511) , Asai, Wu, Wang, Sil, Hajishirzi, 2023-10-17
1. [Agentic retrieval in Foundry IQ (Azure AI Search)](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview) , Microsoft Learn, 2026-10-29

## Frequently asked questions

### Is agentic RAG better than RAG?

For multi-part or ambiguous questions it usually finds better context, because it can check and retry. For simple lookups it mostly adds model calls, latency and token cost, so a single retrieval pass is often the better choice.

### Is RAG still relevant in 2026?

Yes. Agentic RAG extends RAG rather than replacing it: the agent still retrieves documents and grounds its answer in them. Microsoft's agentic retrieval documentation, updated in October 2026, describes the feature as intended for RAG patterns.

### Do you need a vector database for agentic RAG?

No. The retriever is just a tool the agent calls, so it can be keyword search, vector search, hybrid search, a SQL query or an API. Azure AI Search, for example, runs each agentic subquery as keyword, vector or hybrid search.

### What is the difference between agentic RAG and an AI agent with a search tool?

Mostly emphasis. An agent with a search tool is agentic RAG when its main job is answering from retrieved knowledge and it checks what it retrieves. A general agent may search once among many other actions, without grading results.

Services

- [AI agents](https://sdclabs.com/services/ai-agents/)
- [Generative AI](https://sdclabs.com/services/generative-ai/)

Read next

- [How to build AI agents that still work after the demo](https://sdclabs.com/blog/how-to-build-ai-agents/)
- [88% call their data AI-ready, and 43% call it their biggest obstacle](https://sdclabs.com/blog/ai-ready-data/)
- [Is your LLM evaluation measuring your app or your judge?](https://sdclabs.com/blog/llm-evaluation/)

Related terms

- [ReAct agent](https://sdclabs.com/glossary/react-agent/)

[Mykhailo Poliarush](https://sdclabs.com/author/poliarush/) Founder and CEOLeads SDCLabs, an AI development company founded in Krakow in 2011.[LinkedIn](https://www.linkedin.com/in/poliarush/) [GitHub](https://github.com/poliarush)

## Building with AI agents? Talk to an engineer.

[Start a project](https://sdclabs.com/contact/) [Book a call with the CEO](https://calendly.com/sdclabs/60min)
