Pipeline RAG vs Agentic RAG vs GraphRAG: A Reference Decision Guide for Production Systems
We spend a lot of our time at Haal Lab doing autopsies on failed AI projects. Somewhere around mid-2025, the postmortem pattern shifted. It used to be "the model hallucinated" or "the prompt was bad." Now, the majority of the systems we are asked to rescue share a different root cause: the team picked the wrong retrieval architecture for their corpus and their questions. The model was fine. The prompt was fine. The retrieval type was wrong.
This is, in our experience, the number one failure point of agent builds in 2026. A team with a relationship-heavy corpus (contracts, org structures, dependency graphs) deploys vector-only pipeline RAG, watches it fail on every multi-hop question, and concludes they need a bigger model. A team with simple lookup questions deploys a six-step agentic loop, watches latency and cost balloon, and concludes RAG "does not work." Both were retrieval-type errors, not model errors.
This guide is the reference we wish we could hand every client before they write code. We will define the three dominant architectures honestly, compare them on the axes that matter in production, and give you a decision framework you can apply this afternoon.
The three architectures, precisely
Pipeline RAG
The classic: chunk documents, embed chunks, store vectors, retrieve top-k at query time, optionally rerank, stuff into context, generate. One retrieval pass, one generation pass, deterministic control flow. Sometimes enhanced with hybrid search (BM25 plus dense vectors) and cross-encoder reranking — still pipeline RAG, because there is no model in the loop deciding what to retrieve.
- Strengths: deterministic, fast (often under a second end-to-end), cheap, trivially cacheable, easy to evaluate and debug. With hybrid retrieval and a good reranker, this architecture handles a shocking share of real-world questions.
- Failure modes: fails on queries that do not lexically or semantically resemble their answers ("What did the 2024 policy change that the 2022 policy did not?"), fails on multi-hop questions requiring synthesis across chunks, and degrades quietly as the corpus grows because top-k similarity gets noisier.
Agentic RAG
The retrieval loop becomes a tool the agent controls. A grader decides whether the query is well-formed and whether retrieved chunks are relevant; a rewriter reformulates failed queries; a router decides which index or tool to hit; the agent iterates until it judges it has enough evidence, then synthesizes. This is the Self-RAG and Corrective RAG lineage (Asai et al., Sarthi et al.) turned into production control flow.
- Strengths: recovers many wrong-chunk failures. Query rewriting fixes vocabulary mismatch between question and corpus. Relevance grading catches garbage retrieval before it poisons the context. Routing lets one system fan out to multiple indexes, SQL, APIs, and web search. This is the architecture that most naturally supports real agent products.
- Failure modes: variance. The same question can take different paths across runs; a confident-but-wrong rewrite can steer retrieval somewhere worse than the original query. Each extra LLM call multiplies cost and latency, adds a new failure surface, and — critically — makes evaluation harder, because you now need to evaluate the loop, not just the answer. Without eval gates, agentic RAG is a quality lottery.
GraphRAG
Instead of (or in addition to) embedding chunks, an LLM extracts entities and relationships from the corpus into a knowledge graph, often with community detection and hierarchical summaries in the Microsoft GraphRAG style (Edge et al., arXiv 2404.16130). At query time you traverse relationships and read community summaries rather than doing pure similarity search.
- Strengths: the only architecture of the three that natively answers "who/what connects to whom" questions. Global-mode queries over community summaries give surprisingly good whole-corpus thematic answers that chunk retrieval structurally cannot. Entity disambiguation across documents is a graph problem, full stop.
- Failure modes: index construction is expensive (LLM extraction over the entire corpus) and must be re-run or incrementally maintained as documents change. Extraction quality is its own error surface — a mis-extracted relationship is a confidently wrong edge. Graph query answering is harder to evaluate than top-k retrieval, and the operational burden is real: this is a data pipeline, not a vector index you refresh nightly.
The comparison table
| Dimension | Pipeline RAG | Agentic RAG | GraphRAG |
|---|---|---|---|
| Typical end-to-end latency | 300 ms – 2 s | 3 – 15 s | 2 – 20 s (local), higher for global queries |
| Relative cost per query | 1x (baseline) | 3 – 10x | 1.5 – 5x per query, plus large periodic index cost |
| Query-type coverage | Lookup, single-hop | Multi-step, ambiguous, multi-source | Multi-hop, relationship, whole-corpus thematic |
| Determinism / reproducibility | High | Low to medium (loop variance) | Medium (depends on extraction freshness) |
| Eval complexity | Low (retrieval metrics + answer checks) | High (must eval the loop and each gate) | High (graph quality + answer quality) |
| Index maintenance | Embed on ingest, cheap | Same as underlying retrievers | LLM re-extraction, expensive |
| Corpus fit | Fact-shaped documents | Heterogeneous sources, mixed tools | Relationship-heavy corpora |
| Debuggability | Excellent (one trace) | Good but verbose (N decision points) | Hardest (graph + query path) |
| Failure signature | Silent wrong-chunk retrieval | Variance, runaway loops, cost spikes | Stale or wrong edges, thematic drift |
The decision framework
Here is the step-by-step method we run with clients before committing to an architecture. It takes a few hours and saves months.
Step 1: Write down 50 real questions
Not hypothetical ones — actual questions your users asked or will ask, collected from support tickets, SME interviews, or logs. This is the step everyone skips and the step that matters most. Tag each question with its type:
- Lookup — answer lives in one chunk ("What is the warranty period on model X?")
- Synthesis — answer requires combining 2-4 chunks ("Compare the 2022 and 2024 refund policies")
- Multi-hop / relational — answer requires traversing connections ("Who is the ultimate parent company of our supplier's supplier?")
- Whole-corpus — answer is about the corpus itself ("What are the recurring themes in these incident reports?")
Step 2: Compute your question distribution
This single distribution usually decides the architecture:
| Dominant question type (≥60% of questions) | Recommended default |
|---|---|
| Lookup | Pipeline RAG, hybrid retrieval + reranker |
| Synthesis | Pipeline RAG first; agentic escalation on eval failures |
| Multi-hop / relational | GraphRAG, or graph + vector hybrid |
| Whole-corpus thematic | GraphRAG global mode |
| Evenly mixed | Router architecture: pipeline fast path + agentic path + graph index |
Step 3: Characterize the corpus on three axes
1. Shape: fact-shaped prose vs relationship-shaped (contracts, org charts, wikis with dense cross-references). Relationship density above roughly 20-30% of question-relevant content pushes toward a graph. 2. Volatility: how often documents change. High volatility makes GraphRAG's re-extraction cost bite; it barely affects a vector index. 3. Scale: GraphRAG extraction cost scales with corpus size, full stop. A 50k-document corpus with nightly changes is a very different budget conversation than a stable 2k-document one.
Step 4: Baseline before you escalate
Ship the pipeline RAG baseline with hybrid retrieval and a reranker, and evaluate against your 50 questions. We have watched teams assume they "need agentic RAG" only to find a hybrid baseline answers 80% of their tagged questions. Measure retrieval hit rate and answer correctness per question type. Escalate only for the failing categories, and only after you can state the specific failure: wrong chunks retrieved (rewrite/grade helps), wrong source routed (router helps), or relationships missed (graph helps).
Step 5: Gate every escalation with evaluation
This is non-negotiable for agentic systems. Agentic RAG without eval gates adds variance without visibility. Before the agentic loop ships:
- Build a golden set from your 50 questions with graded answers.
- Run it in CI on every prompt, model, router, or index change.
- Track answer correctness, latency distribution (p50/p95, not means), and cost per task.
- Fail the build on regressions, exactly as you would for any other integration test.
The hybrid reality of production systems
In production, we almost never ship a pure architecture. The pattern that works for most of our clients looks like this:
1. A fast path: hybrid pipeline RAG answering lookup questions in under two seconds. 2. An escalation path: an agentic loop (grade, rewrite, retrieve, re-grade) triggered when the fast path's grader flags low retrieval confidence. 3. A graph index for the relationship questions, queried either directly by a router or as one tool inside the agentic loop.
The router itself is a small, cheap classifier — and it gets evaluated like everything else. Router errors are silent quality regressions: a relationship question routed to the fast path returns a confidently wrong answer drawn from similar-looking chunks, which is the worst failure class in RAG because it looks like success to the user.
Failure modes we keep seeing
Being blunt about the ways each architecture bites in production:
- Pipeline: silent degradation as the corpus grows and chunks compete for top-k slots. Watch recall@k over time; it drifts.
- Agentic: loop variance (same question, different answers week to week), runaway loops (cap iterations and retries), and context poisoning when the grader waves through subtly irrelevant chunks. The grader is a model; it is wrong at some rate; measure that rate.
- GraphRAG: stale edges after corpus updates, extraction drift across model versions (re-extract with a new model and your graph silently changes shape), and cost blowouts when nobody budgets the re-extraction cadence against document churn.
References and further reading
1. Edge, D. et al. (2024). "From Local to Global: A GraphRAG Approach to Query-Focused Summarization." arXiv 2404.16130. 2. Asai, A. et al. (2023). "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." arXiv 2310.11511. 3. Sarthi, P. et al. (2024). "Corrective Retrieval Augmented Generation." arXiv 2401.15884. 4. Lewis, P. et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv 2005.11401. 5. Anthropic (2025). "Effective context engineering for AI agents." Anthropic Engineering Blog. 6. Gao, Y. et al. (2023). "Retrieval-Augmented Generation for Large Language Models: A Survey." arXiv 2312.10997. 7. Microsoft Research (2024). "GraphRAG: Unlocking LLM discovery on narrative private data." Microsoft Research Blog. 8. OWASP Foundation (2025). "OWASP Top 10 for LLM Applications."
Retrieval-type selection is a decision you can make deliberately, with data, in an afternoon — and it will matter more to your system's success than any model choice you make this year. Want help auditing a RAG system that is not performing, or designing one from scratch? Contact us — we run exactly the framework above as a structured engagement. For more engineering deep-dives from the studio, browse the blog.