LLM Evaluation Harness for RAG Regression Testing

A regression testing harness for retrieval-augmented generation systems. It turns RAG quality into something a team can review in CI instead of arguing about after a bad answer reaches production.

Why

RAG systems fail quietly. A prompt change can improve tone while reducing faithfulness. A retrieval tweak can look harmless until citation quality slips. I wanted an evaluation loop that made those regressions visible before merge.

Architecture

The harness runs golden-set test cases through a target RAG pipeline, scores outputs with RAGAS and LLM-as-judge evaluators, and publishes the results as CI artifacts. GitHub Actions merge gates block changes when quality falls below configured thresholds.

Prometheus metrics track drift over time: faithfulness, answer relevance, context precision, latency, and cost per run. The point is not to pretend evaluation is perfect. The point is to make quality measurable enough to improve.

What I verified

• Golden-set regression tests for RAG prompts and retrieval changes
• LLM-as-judge scoring across OpenAI and Anthropic models
• GitHub Actions merge gates for quality thresholds
• Prometheus drift tracking for quality, latency, and cost
• Validated against a 4-agent RAG pipeline with roughly 40% hallucination reduction versus a single-shot baseline

What I learned

AI reliability needs software engineering habits: fixtures, thresholds, trend lines, and release gates. The eval harness became the bridge between prompt iteration and production discipline.