Skip to content
All tags

#regression-testing

2 posts
ai guide Ask AI in Practice

Evaluating Ask AI Retrieval: Golden Contracts, Fixtures, Live Runs, and Evidence Boundaries

Ask AI keeps golden contracts, offline fixtures, live SSE output, and production observations separate. A passing fixture proves harness reproducibility; public sources can measure expected-source recall, but they do not expose hidden ranked chunks or establish model-graded faithfulness.

tech deep-dive

Agent Platform Deep Dive (VII)—Evaluation & Quality Gates: Comprehensive Evaluation, Regression Prevention, and an Immune System for Skill Releases

Evaluation is Agent Platform's quality immune system: instead of collecting statistics only after a run, it enforces checks throughout Pre-run, In-run, and Post-run execution. Seven eval categories cover Flow → Step → Skill → Artifact → Evidence → Policy → Regression. A Skill release must pass five gates—Trigger, Functional, Policy, Regression, and Human Review—and any failure blocks it. The Learning Loop moves from Run signals through Proposal, Human Review, Sandbox Eval, Quality Gate, and Publish, under one strict rule: agents propose, humans review, and eval gates decide whether a change can ship.