RAGOps View on GitHub

Open-source release gates for RAG and agents

Stop shipping RAG regressions.

Turn quality, safety, latency, and cost expectations into versioned tests and an explainable go/no-go decision—locally and without a required model API.

Run the 5-minute demo Explore the source
30
Japanese benchmark cases
50+
repository tests
0
credentials for local demo

The problem

AI outputs can look correct and still be wrong.

Teams can demo RAG and agent workflows quickly. The harder question is whether a changed version is safe and useful enough to release.

Without a release contract

  1. Evidence is scatteredNotebooks, vendor dashboards, prompt logs, and manual reviews disagree.
  2. Regressions stay hiddenPrompt, retrieval, and data changes ship without a trusted baseline.
  3. Risk is found after rolloutQuality, policy, latency, and cost failures surface too late.

With RAGOps

  1. Version the expectationQuestions, evidence, thresholds, and red-team policy travel together.
  2. Compare before releaseRun the same scenario against an accepted baseline and candidate.
  3. Leave decision evidenceExport a reviewable report with metrics, failed gates, and case detail.

How it works

From application output to release decision.

RAGOps does not own your model or application. It evaluates portable responses or traces against a versioned release contract.

  1. 01ApplicationYour RAG or agent workflow produces answers, citations, retrievals, and tool decisions.
  2. 02Portable traceJSONL keeps the app, provider, and framework outside the evaluation core.
  3. 03Scenario + checksRun deterministic gates, red-team rules, and optional evaluator plugins.
  4. 04Evidence reportRecord metrics, case failures, findings, budgets, and policy versions.
  5. 05Baseline comparisonMeasure the candidate against the last accepted behavior.
  6. 06PASS or BLOCKReturn a clear decision with reasons—not another dashboard score.

Open-source core: scenario, trace, evaluators, comparison, reports, and release gate. Optional adapters: API, browser workbench, provider integrations, and local control-plane alpha.

Reference demo

See a recorded regression get blocked.

A credential-free Japanese troubleshooting reference app exports the same four cases through Graph+ACL and lexical-only retrieval.

01 · Question

A1000 E-42 のエスカレーション条件は？

Ask when an A1000 E-42 incident must be escalated.

02 · Evidence

Graph + ACL retrieves the policy.

policy-e42-escalation and related machine/incident context remain traceable.

03 · Workflow

Decision: escalate.

The reference agent answers with a citation; consequential external actions still request approval.

04 · Release gate

Lexical-only candidate: BLOCK.

Coverage, precision, and groundedness regress beyond the accepted policy.

Run locally

One command creates a reviewable demo report.

No API key or external service is required.

git clone https://github.com/thangldw/ragops.git
cd ragops
python -m venv .venv && source .venv/bin/activate
pip install -e .
ragops demo --output ragops-demo
Copy the quick start Inspect recorded evidence

Evidence

Two experiments. Two different claims.

The reference deployment proves the integration path. The larger synthetic benchmark validates harness behavior across a wider failure taxonomy.

4-case reference deployment

Graph-assisted baseline passes; lexical-only candidate is blocked.

MetricGraph + ACLLexical onlyDelta
Citation coverage100%75%−25.00%
Citation precision100%75%−25.00%
Lexical groundedness100%78.12%−21.88%
Open reference comparison →

30-case synthetic harness benchmark

Baseline passes; regressed and adversarial candidates fail.

9
failure families
5
adversarial critical findings
2
intentionally blocked candidates

Coverage includes stale evidence, disambiguation, permission leakage, prompt injection, abstention, and consequential action.

Open benchmark report →

These synthetic results validate the harness and this recorded architecture comparison. They do not establish semantic correctness, production security, customer adoption, or business ROI.

Known limits

What remains unsolved.

RAGOps reduces uncertainty by making evidence reproducible. It does not turn synthetic tests into production truth.

Semantic quality

Lexical groundedness is overlap, not entailment or human judgment.

Production identity

The reference ACL is a role-list simulation, not enterprise SSO/RBAC.

Automatic GraphRAG

The reference graph is explicit and small; entity extraction is not automated.

Complete red teaming

Attack metadata exists, but not every attack runs end-to-end against a target app.

Hosted operations

The control plane is a local alpha, not a production multi-tenant service.

Adoption and ROI

Customer-reviewed data, shadow mode, incident ownership, and workflow metrics still need validation.

Current rollout recommendation

Proceed to a customer-reviewed offline pilot. Do not claim production readiness.

Build with us

Reach useful evidence in five minutes.

01Run the demoGenerate a local PASS/BLOCK comparison and HTML report. 02Add a scenarioVersion the questions, evidence, thresholds, and policies that matter to you. 03Write an evaluatorExtend the deterministic core with a focused, testable metric. 04Propose an integrationBring a real workflow and define the acceptance evidence together.

MIT licensed · local-first · provider-independent

Evaluate with evidence. Ship with confidence.

Star or fork RAGOpsBrowse open work