Multi-Agent RAG with LangGraph: when 4 agents beat 1

The honest version of “we built a multi-agent RAG system” usually starts with a single chain that hallucinated 30% of the time and someone in product asking why we couldn’t just “fix the hallucinations.” This post is about what we actually did.

The shape of the problem

We had a single-agent RAG pipeline running over ~50K internal documents. It worked about 70% of the time, which is to say: it failed loud enough for users to file tickets, and quiet enough for those tickets to ship anyway.

Three failure modes dominated:

1. Wrong document retrieved. The query was ambiguous, BM25 + embedding similarity returned the wrong top-k, the model dutifully grounded itself in irrelevant context, and confidently produced a plausible-sounding wrong answer.
2. Right document, wrong synthesis. Retrieval was fine; the model just paraphrased incorrectly or stitched together two unrelated paragraphs.
3. Right document, right synthesis, wrong attribution. Output was correct but cited a paragraph that didn’t actually contain the claim.

A single chain can’t tell which of these is happening. So the fix wasn’t “better prompts” — it was decomposing the chain into agents that each owned one decision.

Architecture

┌─────────┐    ┌──────────┐    ┌──────────┐    ┌──────────────┐
│ Router  │───▶│ Retrieval│───▶│ Verifier │───▶│ Synthesizer  │
└─────────┘    └──────────┘    └──────────┘    └──────────────┘
   │                                                    │
   └────── if low confidence: clarify ──────────────────┘

• Router classifies the query and decides whether to clarify, retrieve, or refuse. Cheap model. ~50ms.
• Retrieval does hybrid search (BM25 + embeddings + RRF) and returns top-k.
• Verifier is the most important agent. Given the query and the top-k, it scores whether the retrieved context actually answers the question. If not, it either triggers a re-query or refuses. This is the agent that prevents quiet wrong answers.
• Synthesizer generates the final response, with citations that the verifier re-checks before returning.

LangGraph made this clean. Each agent is a node, edges encode the routing logic, state flows through. The graph is more declarative than the original LangChain chain and easier to debug — you can replay any single node in isolation.

The eval that actually mattered

Accuracy is the wrong metric. A RAG system can be 90% accurate and ship 10% confident wrong answers, which is worse than 80% accurate with 20% explicit “I don’t know.”

We used RAGAS with three signals:

• Faithfulness — does the answer follow from the retrieved context? (Catches hallucination.)
• Context precision — are the right documents in the top-k? (Catches retrieval failure.)
• Answer relevance — does the answer actually address the question? (Catches off-topic but technically grounded responses.)

The result that mattered: faithfulness went from 73% → 91% when we added the verifier agent. Accuracy barely moved. But faithfulness is what users feel.

What I’d change

A few things I’d do differently if starting again:

1. Eval before architecture. I built the agent graph first and the eval second. Should have been the other way. The eval is what tells you whether the architecture is helping.
2. Cheaper router. I spent too long picking a model for the router agent. It barely matters — almost any model handles “is this a question we can answer” if the prompt is good.
3. Citation grounding earlier. The third failure mode (wrong attribution) was the last thing I caught and the most embarrassing. Should have been part of the eval from day one.

Code

Repo → — full implementation, RAGAS eval pipeline, README with v1 → v2 architecture diff, 3-min Loom walkthrough.

---

If you’re working on production RAG and want to compare notes — eval pipelines, agent boundaries, hybrid retrieval, anything — I’d love to hear from you.