King and Worthington compared 134 paid expert pre-reviews of 124 manuscripts with Claude Opus 4.8 reviews: the GPT-5.5 judge chose the AI review in all 132 decided pairs, and three former journal editors chose it in 26 of 30 judgments. The result covers constructive pre-submission feedback, not journal refereeing.
Evaluate in two layers: did retrieval find the source passage, and is the answer grounded in it and correct. Looking only at answers can fool you: one study found that when the context wasn't enough to answer, models still got 35% to 62% right using their own knowledge, which breaks on questions only your organization can answer. Use order 11's test questions for both layers, and evaluate permission-crossing and unanswerable questions too; calibrate LLM judges against human samples. After launch, turn escalated and complained-about conversations into failure cases, hand them back to document owners, and add them to the test set.
A wiki that people understand can still trip up an AI: what it sees is a few chunks cut out of the pages, without the navigation, heading hierarchy or surrounding text. So before building RAG, write a set of test questions: each with its answer and its source passage, plus a few deliberately unanswerable ones. Azure's RAG guide does this in parallel with content analysis; one study found that evaluating only with AI-generated questions overstated retrieval performance by 29 to 37 percentage points, so mix in real users' questions. If your knowledge base is under about 500 pages, you may not need RAG at all: put the whole thing in the prompt.
To cut the input tokens resent every turn, agent platforms often truncate, clear, or summarize old tool results, and skill instructions, write receipts, and error messages get cut along with them. The Complexity Trap shows observation masking can halve cost, yet Gemini 2.5 Flash with thinking still dropped from 40.4% to 36.4% solve rate. Measure savings against full history and fixes against the current setup, and judge by total cost per successful task, not shorter per-turn input.
Claude's docs say tool-selection accuracy degrades past 30–50 tools, and both Claude and OpenAI now load tools on demand via tool search. But finding a tool isn't finishing the task: on ToolRet's 43k-tool benchmark, no retriever put the complete required toolset in its top 10 more than 35% of the time. This post splits the problem into five failure points, maps seven approaches onto them, and lays out an evaluation.
In a pooled analysis of 50 homework experiments on ASSISTments, hints, breaking a problem into smaller steps, and worked examples beat showing only the answer by d = 0.16–0.27, measured on performance during the homework itself. Interventions that worked in a few courses were scaled up to 247 MOOCs, and none of the main hypotheses replicated. Most course-quality rubrics don't publicly report reliability or validity; in one university's test, none of nine dimensions had a significant direct path to pass rates. The reviews of design-based research all look at how reports are written; none compares whether the materials it produces are more effective.
A chat model is rewarded for a single response people prefer; an agent has to be rewarded for whether a multi-step task actually finished, which needs different data and environments.
Reasoning is the intermediate tokens between input and output; RL fine-tuning beats imitating human solutions because of a reliable verifier, and sampling several times then taking the most frequent answer pushes accuracy further.
AlphaGeometry2 relies on a symbolic engine and AlphaProof on Lean, for a combined 28 points in 2024; in 2025 Gemini Deep Think scored 35 points in natural language, with verification done by human graders.
Laskin's view is that engineers spend 80% of their time understanding systems rather than writing code, and that how far RL goes depends on clean rewards, a problem he considers as hard as superintelligence itself.
A VLA adds an action module to a vision-language model; letting that module's gradients flow back harms language understanding, so knowledge insulation blocks the gradient, while π0.5 co-trains on web data and data from many robots to gain generalization.
Alexandr Wang says Meta listed 100+ behaviors a personal agent needs, built an eval for each, and set two thresholds per row: a launch-blocking floor and a bar for 'good'. They shipped after the first all-green checkpoint and a hands-on check that it was clearly better; they also ran new-user testing. The method is reusable, but post-launch safety reports show all-green is not the same as safe.
An agent needs trustworthy feedback to correct its actions; improving one task’s result does not mean its model weights have learned.
Open-ended evolution changes agent code and workflows; archives preserve branches, while held-out tests assess generalization.
Program candidates need requirement-based counterexamples; speed matters only after correctness checks pass.
Retrieval, trained KV representations, and cache reuse address different bottlenecks; speed does not guarantee freshness.
More candidates increase the chance of a correct solution; credible selection turns that chance into delivered success.
A correct final answer does not validate its derivation; step feedback locates errors, but verifiers still need checking.
Observations, tests, and principle-based critique supply different signals; identify the signal before choosing revision or training.
Distinguish search, parallel execution, and training; subtask artifacts must fulfill the original task criterion.
Useful training feedback must distinguish candidates; filtering, relative advantages, and sampling determine the learning signal.
Candidate search selects solutions; retrieval fills knowledge gaps. Both require specifications and checkable evidence.
Human task duration, artifact quality, and citation support measure different things; evaluate both execution and deliverables.
Diversity, verification, task selection, and efficiency are distinct bottlenecks; experiments need holdouts and falsifiable hypotheses.
Four system design questions that come up often (enterprise RAG, an LLM serving stack, a customer-support agent, online A/B testing with a data flywheel), three Python problems (dynamic batching, sampling, backoff retries plus a token bucket), and a STAR skeleton for behavioral rounds. Each design question has a speakable answer and a self-check list, and all code was actually run.
Seven LLM engineering topics that come up often in interviews: LoRA/QLoRA and fine-tuning vs RAG, RLHF/DPO/GRPO alignment, inference optimization (PagedAttention gives vLLM 2-4x throughput), sampling and decoding, LLM evaluation, the OWASP LLM Top 10 (2026 edition), and hallucination. Each section covers the concept, the mechanism, and how to answer.
Prompt engineering shapes a single instruction, context engineering decides what the model sees right now, and harness engineering wraps the whole non-deterministic system. Bsharat's 26 principles, Breunig's four context failures, and OpenAI's roughly one-million-line, zero-handwritten-code harness experiment each map to one layer. Evaluation is the piece teams skip most, and it belongs in a CI gate that blocks silent regressions.
Lecture 7 of Harvard CS 2881R (Fall 2025) had METR's Joel Becker work through a puzzle. On benchmarks, AI can complete, half the time, tasks that take humans hours, and that length doubles about every seven months. Yet in METR's own randomized controlled trial, experienced open-source developers were 19% slower with AI, and labor-market effects are concentrated among young workers. Becker laid out several reconciliations, centered on benchmark tasks being too clean, scoring too cheap, and human baseliners lacking context. The course had also scheduled frontier safety frameworks (OpenAI's Preparedness Framework, Anthropic's RSP) for this lecture, but they were not covered; this post fills them in from the reading list, showing how they turn capability measurements into thresholds.
A guide to the decoding and evaluation unit in NTHU Prof. Hung-Yu Kao's NLP course (Fall 2025). The first half covers how to pick a word once the model outputs a probability distribution: greedy decoding can't take back a mistake, beam search keeps several candidates but favors short outputs, and top-k / top-p trade determinism for diversity (missing from the slides; the professor covers it verbally in class). The second half covers scoring generated text: BLEU's modified precision and brevity penalty, ROUGE-N and ROUGE-L, perplexity, and what GLUE, SQuAD 2.0, MTEB, and MMLU each measure.
11-768's Assignment 2 has students use one fixed judge, Qwen3-VL-30B-A3B, to flag four error families in every run of a data-visualization agent, graded by the mean MCC across families on a private set (30% of the assignment). The second half packages students' own tasks as Harbor environments, with one wrong solution the verifier rejects and one that fools it. The theme: the grader you write becomes the RL reward later.
Akari Asai's L10 splits deep research agents into three problems: evaluation has to cover four gaps (search difficulty, domain expertise, long-form answer quality, citation support); training runs mid-training → SFT → RL, with DR Tulu's evolving rubrics as the reward for long-form reports; retrieval should let the retriever see the agent's reasoning, which gets AgentIR-4B to 68% on BrowseComp-Plus with Tongyi-DR.
CS224U's fourth unit opens with one question: what can behavioral testing prove, and what can't it? It can never give a guarantee, and when a model fails you first have to ask whether the model or the dataset is at fault. BERT scored 2.2% on negated NLI examples, then 90% after fine-tuning on a small set of them. The unit then covers SQuAD distractor sentences, Breaking NLI, ANLI's human-and-model adversarial collection, and ends with DynaSent's two rounds.
The second half of CS224U's 'NLP methods and metrics' unit skips metric formulas. It asks whether your experiment holds up. Naturalistic or crowdsourced data, adversarial or common cases: the course answers 'both' each time. Lock the test set away. Pick baselines when you write the hypothesis. Compare two models with confidence intervals, Wilcoxon, or McNemar, and run several random initializations. The slides, three videos, and two notebooks are all public. Kawin Ethayarajh's guest session 'Real-world NLP assessments' has no public slides or video.
The first CS224U lecture of Spring 2023 asks "Which U.S. states border no U.S. states?" of every system from Chat-80 (1980) to text-davinci-001. The answers show that the progress is real. The lecture then questions whether that progress counts as understanding, using Levesque's "cheap tricks," models that invent links, and benchmarks that saturate within a year or two. That splits the course map in two: the first half teaches you to build systems with Transformers and retrieval-augmented in-context learning, and the second half teaches you to test them with harder benchmarks, behavioral evaluation, and causal explanation methods.
The CS224U slides compute two numbers from one three-class confusion matrix: accuracy 0.81 and macro F1 0.43. One says the system is good; the other says it gets the two small classes almost entirely wrong. The unit's claim is that different metrics encode different values, and it goes through the bounds, values, and weaknesses of accuracy, the three F-score averages, perplexity, word error rate, and BLEU. Final projects are graded on whether the metrics fit, not on how high the scores are.
CS189 Spring 2026 Lec 11–12 splits classification into two routes. Generative models fit p(x|y) for each class (GDA: shared covariance gives LDA and a linear boundary, per-class covariance gives QDA and a quadratic one). Discriminative models fit p(y|x) directly (logistic regression: sigmoid, softmax, cross-entropy MLE, no closed form, so gradient descent). The bridge: the LDA posterior can always be written in logistic form, but not the other way around. For evaluation, accuracy misleads under class imbalance; ROC/AUC sweeps every threshold and ignores calibration; PR curves care about class balance.
DeepResearch Bench II uses 9,430 expert rubrics covering 132 tasks, and finds that even the strongest agents satisfy less than 50% of criteria. This article breaks down the benchmark architecture, scoring methodology, leaders, and the overall evaluation landscape.
A deep research agent produces a report—maybe thousands of words with dozens of citations. How do you score it? Using LLMs as judges is biased, asking humans is too expensive, and benchmarks can't keep up. STC and other recent approaches try to solve this from the 'confidence' angle—but there's no perfect answer yet.
The first observation of 'What course articles do you have?' was contaminated by an old cache entry. A real cache miss retrieved all four university maps but spent 51.169 seconds across three Writer and Critic passes; after catalog-specific retrieval and review fixes, one uncached production observation passed q21 in 26.821 seconds.
Ask AI keeps golden contracts, offline fixtures, live SSE output, and production observations separate. A passing fixture proves harness reproducibility; public sources can measure expected-source recall, but they do not expose hidden ranked chunks or establish model-graded faithfulness.
Benchmark scores in model releases have three common traps: cherry-picking (only showing wins), contamination (test data leaking into training), and saturation (when everyone scores 90%+, the benchmark stops being useful). The most manipulation-resistant signal is Chatbot Arena's Elo ranking — real humans, blind voting, uncontrolled questions.
Small models don't fail at reasoning first — they fail at format stability: tool-call JSON, diff hunk arithmetic, and context budgets all break. The mature harnesses build evals on real model behavior (pi's model-backed evals, OMP calibrating benchmarks from real session logs, Codex even relaxing its parser for weaker models). looplane picks the narrowest but hardest path: one fixture, five real Ollama runs, a manifest declaring exactly which files and patch fragments count as success — and M2's failure kept verbatim as evidence. Never pass mock off as E2E; never spin partial success into full passes.
None of the five reference projects enforces 'all declared verification commands pass' at the harness level: pi leaves verification to the model, OpenCode and Codex put it in the system prompt, Claude Code uses a separate adversarial verifier subagent but as a soft contract, and only OMP's cleanse actually runs checks from harness code. looplane takes the hardest path: if files changed, every declared verification command must pass before terminal_reason=verified; with no changes, checks don't rerun (no_changes). Whether to verify is decided by code, not by the model.
2023 was the first full academic year after ChatGPT, and LLMs rewrote the NLP conference agenda. ACL's Best Papers examined humor understanding and the propagation of political bias; an EMNLP Best Paper explained in-context learning through information flow; and the HackAPrompt competition paper also won an EMNLP Best Paper award, signaling that security research had entered the mainstream. The year's largest shift was from asking how to make models more accurate to asking how we can tell when a model is misleading us.
You cannot take model vendors' self-reported scores at face value. This guide covers the most important independent evaluation platforms, domain benchmarks, adoption indicators, and official sources in 2026: what each measures, how to read it, where it is biased, and which figures matter for different use cases.
Evaluation is Agent Platform's quality immune system: instead of collecting statistics only after a run, it enforces checks throughout Pre-run, In-run, and Post-run execution. Seven eval categories cover Flow → Step → Skill → Artifact → Evidence → Policy → Regression. A Skill release must pass five gates—Trigger, Functional, Policy, Regression, and Human Review—and any failure blocks it. The Learning Loop moves from Run signals through Proposal, Human Review, Sandbox Eval, Quality Gate, and Publish, under one strict rule: agents propose, humans review, and eval gates decide whether a change can ship.
A Skill is a versioned, installable, and auditable capability package. Its dual-file architecture separates metadata from instructions, explicit binding replaces model-driven routing, and every invocation is recorded. The Learning Loop turns run signals into proposals, sandbox evaluations, human review, and publication while enforcing the principle: agents propose, humans review, and evals serve as the gate.
Phoenix is an MIT-licensed open-source LLM observability and evaluation platform. It collects traces with OpenTelemetry and OpenInference, turns production failures into versioned datasets, compares prompt, model, or RAG changes in experiments, then writes code, human, and LLM evaluator scores back as annotations. It is not Arize AX, and self-hosting defaults require security work.
Lecture 13 separates alignment into specification, distribution shift, oversight, and corrigibility, then uses benchmark selection, leakage, and post-hoc selection experiments to show why a final paper cannot audit an autonomous research workflow.
Lecture 9 begins with log-log linear relationships between data and error, then uses scaling laws to compare architectures, optimizers, batches, and model-data allocations. The Chinchilla dispute shows how fitting methods, observed ranges, and deployment objectives change the answer.
DSPy replaces handwritten prompt strings with task Signatures, execution Modules, and Optimizers that compile better instructions and examples against a dataset and metric.
LangSmith structures LLM applications as projects, traces, runs, and threads, then uses datasets, evaluators, and experiments to turn production failures into offline regression tests. It observes any LLM application and does not require LangChain.
The repository has a 20-query Traditional Chinese/English golden dataset, but no document-level qrels, retrieval runs, raw latency data, or executable benchmark script. Reporting Recall@k, MRR, or nDCG as measured results would therefore be dishonest; this article defines the contract needed to run them reproducibly.
Units 08–12 turn a base model into an interactive system: pre-training establishes capability, post-training shapes behavior, and generation plus evaluation determine how outputs are used.
Units 13–14 connect models to external knowledge; A3 requires data collection, QA annotation, indexing, and ablations under CPU and latency constraints.
Promptfoo plugins generate risk probes, strategies transform attacks, targets execute the system, and graders judge outcomes; useful red teams exercise the full agent application rather than only a foundation model.
CS224U is Stanford's natural language understanding course, the follow-on to CS224N: it covers domain adaptation for sentiment analysis, retrieval-augmented in-context learning, advanced behavioral evaluation and analysis methods, with a bake-off attached to each of the three assignments. The material is a public GitHub repo, and the slides and the full recordings of the online version are public too; the data loading in the first assignment needs a fix before it runs in a fresh environment.
Generating a correct candidate and recognizing it are different capabilities. This series follows the Autumn 2025 syllabus through a course overview and seventeen session guides covering all seventeen content sessions; five guest sessions have no public material and use public material from the speakers on similar topics instead; video Parts 7–9 map to sessions 8, 17 and 20.
Four certifications genuinely test RAG and retrieval evaluation: AWS AIF-C01 (chapters 2 and 3 total 52%, covering RAG, vector stores, and FM evaluation metrics), AWS AIP-C01 (11 of the 27 skill points in its 31% Domain 1 sit in vector storage and RAG), NVIDIA NCP-AAI (Knowledge Integration 10% plus Evaluation and Tuning 13%), and Microsoft AI-500 ('multi-agent RAG architecture' inside its 30–35% Develop area). Google PMLE contributes exactly one LLM-as-a-judge objective, and Claude CCDV-F — the developer certification people most readily assume covers RAG — has no retrieval objective across its eight domains, with Eval at just 2.6%. Includes a same-vendor foundational-vs-professional comparison, a four-vendor terminology map, non-transferable objectives, and a practice project.
A BCG experiment found a jagged frontier: inside it, AI substantially improved consultants' work; outside it, AI made results worse — and people fell asleep at the wheel. The lecture also takes a strong position: avoid fine-tuning wherever possible, because by the time you're done tuning, the next model already beats your fine-tuned version.
Andrew Ng demonstrates error analysis on a deep researcher: columns are the pipeline stages, rows are 10 to 100 queries, you only look at the ones that went badly, and you mark each cell where something broke. The percentages don't have to sum to 100%. He says it takes three or four hours and saves weeks of going the wrong direction — and the fraction of people who actually do it is far below 100%.
Ask a model what a goose looks like to it and it draws a whole flock — because the labeled data tagged a flock as 'goose,' so it thinks the flock is the label. This lecture gives seven ways to open a CNN up, then says honestly: applied to transformers, even the frontier of this research only explains two layers.
Salesforce's number from 20,000 deployments: 90% of the work on an agent happens after launch, the reverse of traditional software. Stripe merges 1,300 PRs a week with no human-written code, and credits the environment rather than the model.
Even with temperature=0, LLM outputs can still fluctuate by up to 15% in practice. To rigorously compare agent changes, you need a frozen golden set, at least 3 runs per query averaged out, LLM-as-judge blind evaluation (pairwise preference flip rate reaches 35%), and paired statistical tests -- not just running each version once and going by feel.
Langfuse is currently the most mature open-source LLM Observability platform. This post covers four core capabilities — Tracing, Prompt Management, Evaluation, and Datasets — showing you how to use them in real projects.
RAG has evolved far beyond simple 'search + generate' into a technology ecosystem spanning ten generations — and since 2025 into an Agentic/Reasoning era. This article is a systematic navigation guide: from Naive RAG to Multi-Agent/LongRAG across ten generations, the post-ten Agentic Era (Search-R1/RL search, MCP, GraphRAG 3.x, vision-native retrieval), retrieval strategies, chunking, embedding, reranking, evaluation frameworks, observability, and cost optimization. Each topic has a dedicated deep-dive article.
No industry standard mandates one RAG evaluation tool. Measure retrieval, generation, and operations separately, then choose Promptfoo, RAGAS, DeepEval, or TruLens for the actual stack.