🌏 中文版
Video status: Recordings require sign-in or course authorization. Source details
Lecture 10 (Oct 26) is a guest lecture by Terminal-Bench co-creators Ryan Marten and Alex Shaw (Laude Institute) on evaluating how well agents complete real tasks in a terminal. The same day, HW2 is released, and its subject is evaluating an agent; HW1 is due Friday, Oct 30.
The assigned readings are the Terminal-Bench project site and Merrill et al.'s Terminal-Bench paper. This post digests the paper, focusing on whether the environment's final state actually meets the requirement after the agent says it is done. The slide guide below follows the PDF page numbers, followed by digests of the assigned readings. Examples and self-checks are teaching exercises, not official homework answers; repeated animation frames, logistics and section dividers are grouped together.
Lecture 10 slide guide
This guest lecture comes from Terminal-Bench co-creators Alex Shaw and Ryan Marten, under the title Evaluating Agents: Terminal-Bench and Harbor. The first half argues that evaluation has become the center of agent development; the second half is their practical experience writing tasks and maintaining a benchmark.
Pages 1–28: Rules are no longer the only source of truth
The speakers start with a short history of software: from numeric machine instructions to assembly to high-level code, humans write rules and programs apply them to new inputs. That is deductive development. Machine learning reverses the direction: humans supply example inputs and outputs, and training infers the rules. The slides then claim all digital technology is becoming inductive: models are learned from data, agents are developed through evaluation, and software is generated by agents.
That sets up the talk's question. The specifier used to be the software engineer; who is the evaluator now? When behavior is learned rather than written, outputs are what you can check. This is also why HW2 asks you to design an evaluation yourself.
Check your understanding: why do the speakers say agents are "developed via evaluation"? Answer criteria: there is no rulebook for an agent's behavior to review line by line; you observe what it does on a set of scored tasks and change it based on the results, so evaluation replaces the specification as the reference for right and wrong.
Pages 29–39: What a Terminal-Bench task looks like
Terminal-Bench starts from the insight that bash is the one tool that covers most work, so every task happens in a terminal. The example on page 38 splits a task into three files. The instruction asks the agent to output a STEP solid model from the 2D drawing at /app/schematic.png; the Dockerfile only provides Python and the image; the tests check that the file is valid and watertight and that volume, surface area, principal inertia, and topology are correct. The tests read the artifact and ignore which commands produced it.
The speakers also show the project's scale (page 36 cites more than 7 million benchmark runs), and on page 39 they shift from "we make the eval" to "you make the eval": Harbor is the tool they built for that.
Check your understanding: why does the STEP example check volume and inertia instead of only whether the file opens? Answer criteria: opening only proves the format is valid; volume, surface area, inertia, and topology correspond to the actual requirement of producing the object in the drawing.
Pages 40–55: Writing tasks: instruction, environment, verifier
Page 42 borrows Anthropic's evaluation vocabulary: a task is one test, a trial is one attempt, a verifier scores it, and a trajectory is the full record. The evaluation harness runs the whole suite; the agent harness lets a model act as an agent. Keep the two apart. Page 43 shows a Harbor task directory, instruction.md, task.toml, environment/Dockerfile, solution/solve.sh, and tests/test.sh, which is agent-agnostic and portable.
The core question in writing a task is what you are measuring and whether you measure it correctly. Instructions should be fully specified, concise, and natural; page 46 contrasts a long original instruction with a shorter version revised after feedback. Environments should use real software and services rather than mocks, with network dependencies baked into the image to reduce flakiness. A verifier is an outcome-based classifier on the environment's final state: think in terms of a confusion matrix, map each check one-to-one to a requirement in the instruction, and use programmatic checks or agent and LLM judges. To prevent reward hacking, isolate the agent environment from the verifier environment and pass only the artifact to the tests; page 52 shows a model searching online for known Lean kernel bugs to pass a grader. The reference solution should mimic the actions an agent would take rather than hardcode the answer, or it will not expose bugs in the environment or verifier.
Check your understanding: the instruction says "write summary.csv," but the verifier also checks column order. What goes wrong? Answer criteria: the verifier demands a condition the instruction never stated, so a correct agent fails, a false negative. Fix it by mapping instruction and verifier one-to-one: state the requirement or relax the check.
Pages 56–64: Evaluation infrastructure and task versions
Evaluation infrastructure has to be reliable, reproducible, fast through parallel independent trials, and portable between local and cloud sandboxes; the same rollout infrastructure can serve testing, training, and production. Tasks need QA too. Page 59 lists static checks, an agent judge, an oracle agent that runs the reference solution, frontier agents, and cheat agents that hunt for loopholes, with a judge reading those trajectories.
The speakers treat tasks as software: they have bugs, users file reports, and new models bring new behavior. Page 61 uses a three-part version number to decide what happens to old results. A change to the instruction, environment, or tools the agent sees bumps the major version and requires a rerun. A verifier-only change, such as fixing an overly strict check, closing a reward hack, or re-tuning a judge, bumps the minor version and regrades stored trajectories. Documentation, pinned dependencies, and typo fixes bump the patch version and reuse results. Every result row should come with full receipts: trajectory, logs, artifacts, and rewards.
Check your understanding: you notice the verifier misses "corrupted the input file" and add the check. Must you rerun last week's 50 trials? Answer criteria: only the verifier changed, a minor-version bump; the stored trajectories and artifacts are enough to regrade without rerunning, provided every trial kept its artifacts.
Reading 1 | Terminal-Bench
Source: Terminal-Bench
TL;DR
Tasks combine instructions, Docker environments, time limits, tests, and reference solutions and score resulting state.
Editorial judgment
| Dimension | Reading judgment |
|---|---|
| Central question | How can a terminal agent's completion claim be verified? |
| Evidence and scope | This guide follows the paper's task definition and version, rather than treating a current leaderboard as course performance. |
Domain background
A terminal can modify extensive state; a successful exit can still leave wrong artifacts or incomplete configuration.
Intermediate guide
Method. The harness starts an isolated environment, supplies instructions and a terminal, and checks resulting state after execution.
For what the guest speakers covered, see the Lecture 10 slide guide above. This section reads the assigned Terminal-Bench paper to see how the paper writes down its task definition. You need to understand tool calls and terminal output; the goal is to write a success condition your agent cannot satisfy merely by saying it is done.
The paper defines tasks through instructions, an isolated Docker environment, tests, a reference solution, and a time limit. The agent chooses its commands. Tests check whether the final container state satisfies the requested outcomes; they do not require the reference command sequence or grade the final chat reply. Bring that distinction back to HW1: using a terminal is an action capability; checking whether the action met the request is an evaluation capability.
Here is an original teaching example, not an official benchmark task or homework solution. Suppose a team asks you to summarize event registrations from this CSV:
city,status
Taipei,confirmed
Taipei,cancelled
Taichung,confirmed
Taipei,confirmedThe request is to count only confirmed registrations by city in summary.csv, without changing the input. Walk through the rows: exclude the cancelled Taipei entry, leaving two Taipei entries and one Taichung entry. The expected output is therefore Taipei 2 and Taichung 1, regardless of row order.
| State after the run | Verdict | Reason |
|---|---|---|
| Says “done,” but creates no output file | Fail | A reply cannot substitute for a file |
| Creates a file, but counts Taipei as 3 | Fail | Cancelled registrations were included |
| Counts Taipei 2 and Taichung 1, but alters the input | Fail | Correct output still violates the input-preservation constraint |
| Correct columns, cities, counts, and unchanged input | Passes this example's conditions | Both the outcome and constraint hold |
A verifier could read the CSV back, compare the city-to-count mapping, and compare the input file's hash before and after execution. This describes a verifier design; it is not a claim that the official harness has been run, or that these checks ensure safety for arbitrary tasks.
Check your understanding: one agent uses Python and another uses shell commands to reach the same valid state. Which should pass? Both: the request did not prescribe a language. What if the verifier only checks that the output file exists? It catches the missing-file case but misses wrong counts and input corruption. Add outcome and constraint checks.
Evidence and result. The paper packages complex command-line work as tasks with environments and tests, enabling outcome scoring beyond agent self-report.
Deeper points
- A reference solution establishes solvability without requiring the agent to follow identical commands.
- Reset state and time limits belong to the task; reusing previous artifacts changes the problem.
Reviewer’s one-line verdict
Outcome verification is closer to task capability than completion claims.
Your take-away
Write a test that inspects the actual artifact and does not accept the final message as proof.
Where it sits in the course
Lecture 10 grounds evaluation in a verifiable final state; Lecture 11 asks how those traces and scores become data. The programmatic graders HW2 asks for can borrow directly from this lecture's verifiers.
Course material for this lecture
- Official slides (PDF): Guest Lecture: Terminal-Bench — Lecture 10
- Assigned readings: Terminal-Bench project site; Merrill et al., "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces" (digested here)
- Schedule: CS329Z site
Course video sources
The official recording policy provides lecture recordings through the course Canvas site. Access requires sign-in. This article uses the public slides, paper, and readings, not a recording or transcript.
Course and recording entries:
Checked: 2026-10-29.
Update log
- 2026-10-29: The series moved from one post per week to one post per lecture. This post now holds Lecture 10 (Mon Oct 26), previously part of the "Week 6: get the data flywheel turning" guide; earlier entries below predate the split.
- 2026-10-29: After the Lecture 10 and 11 slides were released, added page-referenced guides to both lectures with a check per section (writing tasks and verifiers, task versioning, the two forms and five sources of agent data, and what data teaches by accident), and rewrote the opening, course-material map, and Terminal-Bench reading where they called the lectures upcoming.
- 2026-10-23: Rebuilt the guide around source status, mechanisms, a worked scenario, and answer criteria; separated detailed assigned readings from lecture preparation.
- 2026-10-23: Added the announced Terminal-Bench guest readings, an original outcome-verification example, a bilingual diagram, and development/test separation. The guest session is upcoming; this is a reading guide, not a talk transcript.
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-09-29: HW1 was restructured (no more from-scratch / framework split, the corpus is now a corporate email archive) and the midpoint demo became a recorded video due Nov 4 at 11:59 p.m.; rewrote the timeline, trace-logging, and course-position passages to match
- 2026-10-10: Rechecked video status. The official Logistics page says recordings require logging into the course Canvas, opening the entry confirmed the sign-in wall, and no public recording was found; the status is now sign-in required.
References
- On this site: Lecture 9: Optimization, Lecture 11: Data for Agentic Systems, Stanford CS329Z course overview
- Course: CS329Z schedule
- Course material (Mon Oct 26): slides