RAGOps case study — Thang Luu

01 · Production problem

A model-quality change needs a release decision, not another dashboard.

RAG and agent releases can improve an aggregate score while breaking a language, source type or safety slice. Teams then debate whether a candidate is safe to ship, while the benchmark, policy and final decision live in different tools.

RAGOps addresses that failure class by comparing recorded candidate traces with an accepted baseline, applying a versioned policy, and emitting one canonical PASS, WARN or BLOCK decision.

02 · Architecture and trade-offs

Keep the decision core offline; treat integrations as adapters.

Portable scenario and trace contracts enter a dependency-free Python core.
Deterministic evaluators compute quality, safety, slice and distribution evidence.
A versioned release policy resolves findings into one canonical decision.
JSON, Markdown, HTML, JUnit, SARIF and evidence bundles render the same decision.

The core does not import FastAPI, Uvicorn or vendor SDKs. API, workbench and external evaluation systems sit behind optional adapters. This makes a release result reproducible without credentials or network access, at the cost of requiring recorded traces rather than live production observation.

Architecture · Contracts · Operations

03 · Benchmark or failure case

The flagship fixture fails visibly and reproducibly.

The credential-free demo intentionally produces BLOCK: citation coverage drops from 1.0 to 0.5, citation precision from 1.0 to 0.5, and lexical groundedness from 1.0 to 0.6. The v2.0.2 Failure Zoo also includes a synthetic retrieval-poisoning specimen that blocks unsupported citations, unsupported claims and forbidden output terms.

RAGOps demo report: citation coverage and precision fall from 1 to 0.5; groundedness falls to 0.6; four gates fail. — Synthetic regression report from RAGOps v2.0.2. The report shows FAIL; the canonical release decision is BLOCK. Screenshot reproduced on 4 October 2026; the generated fixture uses a fixed timestamp.

Evidence type: reproducible synthetic regression. It proves the evaluator and policy response to controlled failures; it does not prove production adoption or real-traffic performance.

The exact default-branch revision 4b59aec passed public CI and CodeQL.

04 · Reproduce in under five minutes

Run the released package without credentials.

python -m pip install ragops==2.0.2
ragops demo --profile engineer --output ./demo-output
ragops evidence verify --bundle ./demo-output/evidence
open ./demo-output/release-report.html

Expected result: the report shows a synthetic BLOCK decision and the evidence verifier accepts the generated bundle. On Windows, use Start-Process for the HTML report.

05 · What I designed and led

I designed the product boundary, canonical decision model and release evidence path.

I own the versioned contracts, deterministic core, policy semantics, evidence bundles, adapter boundary, CI formats, governance and release qualification. As maintainer, I also reviewed external work for retrieval-poisoning coverage and a Phoenix adapter, requiring provenance, semantic and fail-closed error-boundary corrections before merge.

06 · Limitations and missing evidence

The deterministic release path is implemented; external validity remains open.

No consented production pilot or customer adoption evidence is published.
No public non-synthetic evaluation result is published yet.
Recorded traces do not replace live observability or incident response.
Lexical groundedness is a deterministic proxy, not a semantic judge.
The repository ships no hosted MCP connector; API and workbench are local optional components.

The production-pilot runbook defines what new evidence must contain before those claims can change.

RAGOps

Recorded traces → release decision