Evaluating Ask AI Retrieval: Golden Contracts, Fixtures, Live Runs, and Evidence Boundaries

🌏 中文版

Optional companion reading: Beginners can read this article directly. For extra context, see RAG Evaluation Frameworks and Tool Selection and RAG A/B Testing.

RAG evaluation often starts with an attractive score table that does not say whether its inputs came from fixtures, development, or production. Ask AI resolves the evidence kind before scoring: a golden contract states expectations, an offline fixture checks the harness, a live runner captures public SSE, and production observations retain separate deployment and operator evidence.

This article follows the Ask AI evaluation runbook. You can copy its directory structure, commands, and artifact boundaries, but it will not manufacture raw ranked chunks that the public API never exposed.

Keep four evidence layers in separate buckets

rag-golden-dataset.json currently uses schema 1.0 and case IDs q01 through q21. IDs and existing expectations are regression identifiers and should not be renumbered when wording changes.

The fixture contains independent candidate output for q01–q04 and q21 only. It does not pretend to call Ask AI and is not a production sample. The dataset adapter validates schema, unique IDs and queries, and retrieval-contract fields.

Start offline to verify the measuring tool

The baseline fixture writes candidate output, per-case scores, public-trace placeholders, and a summary report:

pnpm eval:rag:fixture

Default output:

.work/rag-evals/fixture/
├── baseline-outputs.jsonl
├── baseline-scores.jsonl
├── baseline-traces.jsonl
├── baseline-report.json
└── promptfoo-report.json

Test Promptfoo's dataset adapter, provider, and assertion before evaluating its fixture:

pnpm test:promptfoo
pnpm eval:promptfoo:fixture

A passing fixture means the fixed candidate satisfies the current contract. It catches schema drift, duplicated expectations, scorer regressions, and broken Promptfoo wiring. It does not show that D1 or Vectorize retrieved the same material, and it does not test current model generation.

A live runner must parse SSE as a stream

After starting Astro with normal Ask AI bindings, provide an authenticated admin cookie:

RAG_EVAL_COOKIE='admin-session-cookie' pnpm eval:rag
RAG_EVAL_COOKIE='admin-session-cookie' pnpm eval:promptfoo

The default target is http://127.0.0.1:4321; override it with RAG_EVAL_BASE_URL. Live requests set traceScope: eval and cacheMode: bypass. The API grants those capabilities only to an admin and skips both semantic-cache reads and writes.

eval-rag-baseline.mjs uses a stream reader to parse token, sources, related, agent_step, done, and error. Treating SSE as ordinary JSON with response.text() loses event boundaries and behaves poorly for long streams.

Live artifacts go under .work/rag-evals/live/, separate from fixtures. To compare pipeline engines:

RAG_EVAL_COOKIE='admin-session-cookie' pnpm eval:rag:matrix

Only RAG_EVAL_ENFORCE=1 makes aggregate threshold failures exit non-zero. These are local evaluation entry points. They do not by themselves prove that scheduled CI or a production deployment succeeded.

Interpret the baseline metric names by their implementation

The baseline reports contextRecall, answerRelevance, and faithfulness, but their names must not outrun the data.

contextRecall measures how many expected source locators appear in public sources. The API does not expose hidden retrieved context, so this is not chunk-level context recall.

answerRelevance uses token coverage over expected answer points and applies a forbidden-claim gate. It is useful for deterministic regression, not a substitute for human semantic review.

faithfulness combines two observable checks: whether displayed sources align with expected sources and whether answer citation URLs belong to the displayed source set. Forbidden claims force failure. This is a citation and source-alignment heuristic, not a model judgment of whether every sentence is entailed by evidence.

baseline-traces.jsonl records public agent_step events only. It must not be described as a complete retrieval or model trace.

q21 shows when a retrieval contract is justified

An ordinary golden case can stop at answer points, expected sources, and forbidden claims. Add retrieval_contract only when the public response exposes enough signals to enforce it.

The q21 question “What course articles do you have?” checks that:

• Stanford, MIT, CMU, and Berkeley course maps all appear.
• Two unrelated Cloudflare posts appear in neither sources nor answer.
• At least four sources are distinct.
• Latency remains within thirty seconds.
• done.cached is false.

Promptfoo does not duplicate those values. promptfoo-tests.mjs builds tests from the golden dataset and hands the contract to a deterministic assertion. Change q21 in the dataset only, so baseline and Promptfoo do not drift into separate truths.

Add a case from a new incident

Suppose another Chinese short query returns no sources. Preserve a publicly reproducible query, then define the contract:

1. Assign a stable ID that will not be reused.
2. Describe the answer points a reader needs; do not freeze one model's exact wording.
3. Add only expected sources that public sources can verify.
4. Add claims or sources that previously leaked into the answer as forbidden items.
5. Add retrieval_contract only when latency, cache, and source uniqueness are publicly observable.
6. Create an independent fixture candidate; do not copy live output into a fixture and call it independent validation.

After editing the dataset, run:

pnpm exec node --test \
  evals/rag/adapters/golden-dataset.test.mjs \
  evals/rag/provider.test.mjs \
  evals/rag/assertions.test.mjs

pnpm exec node --test scripts/eval-rag-baseline.test.mjs

Then run fixture and live checks. Use RAG_EVAL_ARTIFACT_ROOT to relocate all artifacts, or override individual report, output, score, and trace paths.

The q21 production evidence stops at the public contract

After retrieval-v3 deployed, one uncached q21 observation passed the four required sources, zero forbidden sources, source uniqueness, and latency contract in one public stage sequence. That record supports the public contract for that request and deployed version.

The repository does not contain sanitized raw live output, score, or trace artifacts from the observation, so the 26.821-second measurement and eight checks cannot be recomputed from git alone. It contains no raw ranked chunks, complete Writer context, or every Critic field. The observation is not a long-term latency benchmark and is not model-graded faithfulness.

An independently auditable production report should preserve scrubbed baseline-outputs.jsonl, baseline-scores.jsonl, baseline-traces.jsonl, and dataset identity. User content and session cookies require data minimization and secret scrubbing before any artifact is retained.

Promotion gates should read artifacts, not impressions

Before promoting a retrieval change, verify that:

• Dataset schema and case IDs have not drifted.
• The fixture harness passes.
• A live run records cached: false and writes to the live artifact directory.
• Aggregate thresholds pass for the selected engine.
• Incident-specific retrieval contracts pass.
• Public sources and agent steps are not described as hidden context or a full trace.

The useful part of this workflow is not another score. Every score retains an evidence kind, dataset ID, and artifact path. When the API does not expose a signal, say that it was not measured; expand the API or trace contract only when the next investigation requires it.

Positioning Against Standard Table QA Benchmarks

Ask AI's q01–q21 is a task-specific evaluation: fixed queries, expected sources, and forbidden claims, designed to catch regressions in this particular system. The academic community maintains a separate lineage of general-purpose table QA benchmarks that measure cross-dataset generalization:

• WikiTableQuestions (Pasupat & Liang, 2015): 22,033 query–table pairs. The foundational table QA benchmark and still a standard leaderboard fixture.
• HybridQA (Chen et al., 2020): Multi-hop QA across tables and free text, forcing pipelines to combine structured lookup with semantic retrieval.
• TableRAG (arXiv:2506.10380, 2025): A recent heterogeneous-document reasoning framework evaluated on HeteQA, HybridQA, and WikiTQ, measuring how well RAG handles tabular content.

The two evaluation styles are complementary, not competing: task-specific contracts detect behavioral regression; standard benchmarks measure generalization. If Ask AI grows to handle more structured content (tables, lists, comparison matrices), incorporating a WikiTableQuestions subset into the golden dataset is a viable next step.

Update Log

• 2026-09-03: Added positioning against standard Table QA benchmarks (WikiTableQuestions, HybridQA, TableRAG)

References

• Ask AI RAG evaluation runbook
• Golden dataset q01–q21 and q21 retrieval contract
• Independent offline fixture
• Golden dataset adapter
• Promptfoo test generation
• Baseline live and fixture runner with deterministic scoring
• WikiTableQuestions — Pasupat & Liang (2015), foundational table QA benchmark
• HybridQA — Chen et al. (2020), multi-hop QA across tables and free text
• arXiv:2506.10380 — TableRAG: A RAG Framework for Heterogeneous Document Reasoning (2025)