Case study
Multi-Agent RAG Orchestration
A 4-agent retrieval-augmented system on LangGraph with RAGAS evaluation. Faithfulness 73% → 91% over a single-chain baseline.
Stack
A production retrieval-augmented system that decomposes a single LLM chain into four specialized agents — router, retriever, verifier, synthesizer — connected via a LangGraph state machine.
Why
A single-chain RAG pipeline has no way to distinguish between three different failure modes: wrong document retrieved, right document but wrong synthesis, and right document right synthesis but wrong citation. Decomposing the chain into agents that each own one decision lets you measure and fix each one independently.
Architecture
The graph: query enters the router (cheap classification model), goes through retrieval (hybrid BM25 + embedding + RRF over a 50K-document corpus in pgvector), then a verifier scores whether the retrieved context actually answers the query (re-query or refuse if not), then a synthesizer generates the response with citations that the verifier re-checks before returning.
Eval
RAGAS on a 200-query golden set, run nightly in CI. Three signals: faithfulness, context precision, answer relevance. Prometheus dashboards for production drift.
The headline number: faithfulness 73% → 91% vs the single-chain baseline. Accuracy moved less; what users actually feel is faithfulness.
What’s in the repo
- The LangGraph state machine, with each agent as a node
- Hybrid retrieval pipeline (BM25 + dense + RRF)
- RAGAS eval harness with the 200-query golden set
- Prometheus metrics + Grafana dashboard JSON
- README with v1 → v2 architecture diff
- 3-min Loom walkthrough
← all work