Agentic RAG: Letting the LLM Decide When to Search Again

TL;DR For complex multi-hop questions, a single RAG search isn't enough. Agentic RAG lets the LLM evaluate whether retrieved results are sufficient — if not, it rewrites the query and searches again, forming a ReAct loop.

#rag #agentic-rag #react #multi-hop #llm-agent

Table of Contents

The ReAct Loop
The Decision Prompt
Trigger Conditions
How It Differs from CRAG
Multi-Hop Reasoning in Practice
The Takeaway
Changelog
References

Series: The RAG Techniques Compendium (21 / 48)

🌏 中文版

Standard RAG is a single-pass pipeline: query → retrieve → generate. That works fine for most questions, but falls apart when a query requires multi-hop reasoning. Recent systematic surveys (Singh et al. 2025, Liang et al. 2025, Mishra et al. 2026) have codified this class of approaches — where the LLM drives retrieval decisions — as Agentic RAG: an architectural paradigm that moves from single-shot retrieval to multi-turn reasoning-driven search.

"Plan me a climbing trip leaving from Taichung — intermediate level, doable on a weekend, with routes at different grades so everyone in the group can climb."

This question spans several dimensions:

What climbing crags are near Taichung? (geography)
What grade distribution does each crag offer? (route info)
What grade range counts as intermediate? (skill assessment)
How accessible are they on weekends? (logistics)

One search can't cover all of that at once. Agentic RAG lets the LLM evaluate whether the current context is sufficient during execution — and if it isn't, decide what to search for next.

The ReAct Loop

ReAct (Reasoning + Acting) is the core pattern behind Agentic RAG:

Reason: Evaluate current context, decide next step
Act:    Execute the decision (search / answer / broaden)
Observe: Receive search results, update context
Reason: Re-evaluate... (loop)

Here's how it looks in the implementation:

async function agenticRetrieve(ctx: PipelineContext): Promise<void> {
  let step = 0;

  while (step < ctx.config.agentic_max_steps) {
    const candidates = ctx.candidateMatches;

    // Check if we have enough
    if (candidates.length >= ctx.config.agentic_min_docs_to_answer) {
      ctx.agenticDecision = 'ANSWER';
      break;
    }

    // LLM decides: rewrite query / broaden filter
    const decision = await agentDecide(ctx.currentQuery, candidates, ctx.config);

    if (decision.action === 'RETRIEVE') {
      // Re-run search with rewritten query
      ctx.currentQuery = decision.rewrittenQuery;
      const newResults = await hybridSearch(ctx);
      mergeResults(ctx, newResults);
    } else if (decision.action === 'BROADEN') {
      // Relax filter constraints
      ctx.vectorFilter = relaxFilter(ctx.vectorFilter);
      const newResults = await hybridSearch(ctx);
      mergeResults(ctx, newResults);
    } else {
      break; // ANSWER
    }

    step++;
  }
}

agentic_max_steps prevents infinite loops. The default is 3 steps and can be tuned.

The Decision Prompt

The agent's decision LLM receives:

Current query: {query}
Documents found so far ({n}): {document_summaries}

Choose one:
ANSWER   — Context is sufficient; generate a response
RETRIEVE — More information needed; rewrite the query (provide new query)
BROADEN  — Filter constraints are too strict; relax the search scope

The LLM returns a structured decision:

{
  "action": "RETRIEVE",
  "rewrittenQuery": "Taichung climbing crag transportation options",
  "reasoning": "The current documents lack transportation info; need to supplement."
}

Trigger Conditions

Agentic RAG isn't on by default. It requires:

rag_strategy === 'agentic' or rag_strategy === 'auto' (in auto mode, the strategy is chosen based on queryType)
queryType === 'complex'

The reason is straightforward: Agentic RAG has significantly higher latency than standard RAG (multiple LLM calls + multiple searches), so it's not appropriate for every query. Lin et al. (2025) compared one-shot and iterative retrieval head-to-head, finding that multi-round iteration significantly improves recall on multi-hop questions — but for simple queries, the extra steps introduce noise. This validates an "activate only when needed" strategy.

Rough magnitudes measured on this system (they shift with the model, the retrieval backend, and the step count — these are not universal figures):

Standard RAG: 5–8 s
Agentic RAG:  10–20 s (depending on number of steps)

Are users willing to wait longer in exchange for a more complete answer? That depends on how complex the query is. auto mode lets the system make that call. Another concern worth noting is search efficiency: agents can issue redundant or sub-optimal queries. Wu et al. (2025) showed at EMNLP 2025 that reducing uncertainty can cut unnecessary retrieval steps while maintaining answer quality.

How It Differs from CRAG

CRAG is a rule-based fallback triggered by zero results; Agentic RAG is LLM-driven intervention when results exist but aren't good enough:

	CRAG	Agentic RAG
Trigger	Zero candidate documents	LLM judges context insufficient
Decision	Rule-based (remove filter)	LLM (rewrite query / broaden)
Complexity	Low	High
Added latency	~+0.5 s (one extra search)	+5–15 s (multiple LLM calls)

The two can run together: CRAG as a baseline safety net, Agentic RAG as the high-quality path.

Multi-Hop Reasoning in Practice

For queries that require synthesizing multiple sources, Agentic RAG clearly outperforms standard RAG:

Standard RAG: search "Taichung climbing trip" → retrieve a few crag writeups → LLM generates limited suggestions from that sparse context

Agentic RAG:

Step 1: Search "crags near Taichung" → finds Dakeng, Guguan
Step 2: LLM notices grade info is missing → searches "Dakeng crag route grades"
Step 3: LLM notices logistics are missing → searches "Dakeng crag how to get there"
Synthesizes all three passes → produces a thorough trip plan

Each step fills a specific gap in the context rather than throwing one broad search at the wall and hoping for the best.

The Takeaway

Agentic RAG represents the evolution of RAG systems from passive retrieval to active reasoning. It's not suited for high-traffic, latency-sensitive scenarios — but for complex planning and multi-hop reasoning queries, the quality improvement is substantial. For a detailed comparison of multi-hop architectures, see Multi-hop Retrieval: When the Answer Is Scattered Across Multiple Documents; for a lighter alternative that doesn't require a full agent loop, see Self-RAG: Letting the Model Decide When to Retrieve with Reflection Tokens. The InfoDeepSeek benchmark (Xi et al. 2025) confirms this: on information-seeking tasks that require synthesizing multiple sources, agentic strategies achieve significantly higher completeness than single-pass retrieval.

The core design principle: give the LLM enough information instead of making it guess. Rather than asking the model to reason from an incomplete context, let it run a few more searches until it has what it needs. Agentic RAG hands that judgment back to the LLM.

Changelog

2026-09-03: Added 6 Agentic RAG survey and evaluation papers to strengthen academic grounding
2026-08-19: Fact-checked against primary sources and refreshed; perishable details handed back to official docs. Added to the "RAG Techniques Compendium" series.