Run the RAG eval before you redesign the architecture

The mistake I made on my first serious RAG build was subtle: I treated evaluation as proof that the architecture worked, instead of using evaluation to decide what the architecture should be.

That sounds small. It is not.

When a RAG system gives a bad answer, people tend to reach for architectural language quickly: add an agent, add a verifier, add re-ranking, add a fallback model, add a better prompt. Some of those changes help. Some just make the system harder to debug. Without evals, you are guessing with better vocabulary.

The useful question

The question is not “should this be multi-agent?”

The useful question is:

Which failure mode is happening often enough that it deserves its own control point?

In the system I worked on, the common failures were:

1. The retriever found the wrong documents.
2. The retriever found the right documents, but the model synthesized the answer badly.
3. The answer was mostly right, but the citation did not support the claim.

Those are different problems. A single aggregate score hides them. A prettier agent graph also hides them if you do not measure each boundary.

What I would measure first

Before changing architecture, I would create a small golden set and score three things:

• Faithfulness: Does the answer actually follow from the retrieved context?
• Context precision: Did retrieval surface documents that answer the question?
• Answer relevance: Did the response address what the user asked?

RAGAS is useful here, not because it is perfect, but because it gives you a repeatable starting point. The goal is not to outsource judgment to a metric. The goal is to make regressions visible enough to discuss.

Where agents helped

The verifier agent was valuable because the eval showed a specific pain: answers were often fluent before they were grounded. Once that was visible, the architecture became obvious. Put a checkpoint between retrieval and synthesis. If context does not answer the question, re-query or refuse.

The router was less dramatic. It made the graph cleaner, but it was not where quality moved. That was a useful thing to learn because it kept us from over-investing in the wrong part of the system.

The release habit

The real unlock was treating RAG quality like normal software quality:

• Golden-set cases live with the code.
• Prompt and retrieval changes run in CI.
• Quality thresholds block risky merges.
• Drift metrics show whether production behavior is changing.

That habit matters more than the specific model, framework, or graph layout.

The takeaway

Do not start by asking how many agents your RAG system needs.

Start by finding the failure mode you can prove. Then add the smallest architectural boundary that makes that failure mode measurable, debuggable, and harder to ship.