Two reports that both say "attack success rate 5%" cannot be compared directly: behind the number sit who set the questions (the attacker model), who judged (the judge, a model or rule that decides), and what the denominator counts. garak's own FAQ says its scores have no scientific validity, and in HackAgent's white paper the order of two models flips with the judge threshold. I found no independent accuracy evaluation of any of the four.
Ask AI keeps golden contracts, offline fixtures, live SSE output, and production observations separate. A passing fixture proves harness reproducibility; public sources can measure expected-source recall, but they do not expose hidden ranked chunks or establish model-graded faithfulness.
Promptfoo combines prompts, providers, test cases, and assertions in YAML to produce repeatable local and CI evaluation matrices, with red teaming against the same targets. It lowers the testing barrier but does not remove output variance, LLM-judge bias, or hosted data-flow concerns.
Promptfoo plugins generate risk probes, strategies transform attacks, targets execute the system, and graders judge outcomes; useful red teams exercise the full agent application rather than only a foundation model.
No industry standard mandates one RAG evaluation tool. Measure retrieval, generation, and operations separately, then choose Promptfoo, RAGAS, DeepEval, or TruLens for the actual stack.