CME295 Lecture 8 starts from the fact that human rating is slow and expensive and BLEU/ROUGE can't recognize a paraphrase. It covers how LLM-as-a-Judge works, three biases (position, verbosity, self-enhancement) and six best practices, splits agent failures into tool prediction, tool execution and response generation, and closes with what MMLU, AIME, SWE-bench, HarmBench and τ-bench each measure, plus pass^k and Goodhart's law.
CME295 Lecture 6 breaks reasoning models into three pieces: emit a reasoning chain before the answer, run RL on verifiable rewards like "is the answer correct," and use GRPO, which takes the group's average reward as the baseline instead of training a value model. RL alone took DeepSeek-R1-Zero from 15.6% to 71.0% pass@1 on AIME 2024, and distilling R1's traces into Qwen-32B beat running RL on the 32B model directly.
The 2026 edition of CME295 Lecture 5, "LLM systems" (October 30), lists seven topics: distributed training, inference optimizations, KV caching, speculative decoding, efficient kernels, FlashAttention, and hardware trade-offs. Written before the lecture, this post uses about 70 slides from the 2025 Lectures 3 and 4 plus the original papers to tie them into a single ledger: an H100 needs roughly 295 operations per byte moved to saturate its compute, while token-by-token generation does about 1 per byte of weights read, so most speedups are about moving less data.
CME295 Lecture 4 splits LLM training into two stages: pretraining on trillions of tokens (Llama 3 used 15 trillion), then SFT on thousands to millions of demonstrations so the model stops continuing text and starts answering. In between sits a map of memory savers (ZeRO, FlashAttention, mixed precision); the lecture closes with LoRA and QLoRA, which let people without big GPUs finetune, with QLoRA cutting VRAM by about 16x on a 65B model.
SFT only teaches a model to imitate good answers; it has no way to say which answers are unacceptable. CME295 Lecture 5 covers how to collect preference pairs, walks through the two steps of RLHF (a reward model trained on roughly 10,000 human labels, then PPO on roughly 100,000 examples), and ends with DPO, which folds the whole RL pipeline into a single supervised loss. The 2026 edition splits this lecture between Lecture 3 (training) and a new Lecture 4 (reinforcement learning).
The 2026 CME295 Lecture 4 (October 16) gives RL its own lecture, with seven syllabus items: mathematical conventions, reward design, policy gradients, limitations, PPO, GRPO, and on-policy distillation. This post walks the math ahead of class: start from ∇log π times a score. SFT uses a score of 1, PPO estimates it with a value model, GRPO uses the group mean, and on-policy distillation uses the teacher's per-token log-prob gap. In the Qwen3 report, starting from the same checkpoint, RL reached 67.6 on AIME'24 with 17,920 GPU hours, while on-policy distillation reached 74.4 with about 1/10 of that (1,800 hours).
CME295 Lecture 1 threads a single sentence, "A cute teddy bear is reading.", through the whole class: split it into tokens, turn them into vectors, see why an RNN can't hold on to long sentences, then translate it into French with self-attention and an encoder-decoder. The 2026 edition drops the entire section on NLP tasks and evaluation metrics and opens instead with a timeline running from 2017 to the agent era.
CME295 Lecture 2 takes the original Transformer apart and refits it: position information moves from "added to the embedding" to RoPE's "rotate Q and K inside attention"; attention gets cheaper with sliding windows and MQA/GQA; models split into encoder-only, encoder-decoder, and decoder-only families; and the second half dissects BERT's MLM (15% of tokens) and NSP pretraining. The 2026 edition folds all of this into a single "Large Language Models" lecture, and BERT is no longer a syllabus item.
CMU 11-768's first assignment starts from an empty ReAct loop: a bash-only CodeAgent fixes a bug in a chess app, context compaction is added to solve a SWE-bench task, and the same loop becomes a ChessAgent that runs a two-ply search with simulate_move, run_python, and a skill. All 100 points are graded by replaying submitted patches and trajectories offline.
11-768's Assignment 2 has students use one fixed judge, Qwen3-VL-30B-A3B, to flag four error families in every run of a data-visualization agent, graded by the mean MCC across families on a private set (30% of the assignment). The second half packages students' own tasks as Harbor environments, with one wrong solution the verifier rejects and one that fools it. The theme: the grader you write becomes the RL reward later.
Lecture 1 of 11-768 strips an agent to its minimum: tool definitions and tool calls are just tokens, the harness parses, executes, and feeds results back into context, and running a ReAct loop makes it an agent. Neubig then lists six capabilities a good agent needs, each of which can be built through training or through the harness, and argues that an agent is a system of harness, sandbox, inference, training, and monitoring — not just a model.
Neubig splits tool use into five layers: capabilities, mechanics, constraints, interfaces, and systems. A tool call is just tokens the model emits; the harness parses, validates, and matches results back by call ID. Constrained decoding guarantees form, not correctness. MCP's real value is credential brokering. And the same model served by different providers can swing from roughly 15% tool-call errors to under 0.1%.
An agent resends its whole history on every call, so five calls already add up to 80K input tokens; 1,500 OpenHands sessions averaged 78K tokens, 37% of them tool results. Neubig works on two layers: at the model layer, hybrid attention (many local layers, one global) plus length curricula make million-token context possible; at the harness layer, stable prefixes earn cache reads roughly ten times cheaper, and compaction that keeps anchors and externalizes evidence gets past the limit — evaluated by how the agent continues afterwards.
Lecture 4 of CMU 11-768 sorts cross-task experience into episodes, facts, and skills, stored as external artifacts rather than in context or weights. Human-written skills load through SKILL.md and progressive disclosure, lifting the average SkillsBench pass rate from 33.9% to 50.5%. Skills an agent induces itself can be tested before admission when written as code, but break easily on a new website. The hard part is the lifecycle: imperfect judges, over-retrieval, and bloated skill libraries each eat into the gains.
Lecture 5 of CMU 11-768 defines an agent's plan as an explicit, inspectable, revisable representation of intended behavior for this task, and gives four reasons to add planning structure: modularity, environment feedback, long horizons, and control. Fried's own MACU has a manager decompose tasks into a DAG and dispatch parallel sub-agents, raising Odysseys success from 8.5% to 34.0%; on an OSWorld subset, no planning scores 25.0%, an initial DAG with no revisions scores 27.8%, and allowing 10 revisions reaches 58.3%.
Neubig's L6 splits coding agents into three layers: train a model that can code (pre-training, mid-training, infilling, RL from test rewards), wrap it in a localize–edit–verify loop with the right editing tools so it can change a repo, then evaluate and train it in SWE-bench-style executable environments. Fixing bugs is only about 15% of a developer's day; the next frontier is tests, CI, and maintenance in the outer loop.
JY Koh breaks computer use agents into three questions: evaluation has moved from single clicks (ScreenSpot-Pro, Mind2Web) to programmatic end-state checks (WebArena, OSWorld), VLM judges, and long-horizon rubrics (Odysseys, OSWorld 2.0); the model is a VLM reading interleaved screenshots and actions; training runs pre-training for grounding → SFT on human and synthetic trajectories → RL in resettable simulated environments.
Yueqi Song breaks agent SFT into six decisions: compute loss on assistant tokens only (including the stop token); choose trajectories carefully (runs that pass tests can still teach bad habits, and switching teachers or adding new tasks beats sampling more); unify formats with the Agent Data Protocol; watch packing and template drift during training; evaluate in the real harness; and pick the SFT checkpoint for the RL that follows, not for its own best score.
Using a guess-a-number-from-1-to-16 game, Daniel Fried frames RL as an extension of SFT: he names SFT's three gaps (task mismatch, no learning from failures, never seeing its own mistakes), then derives ReST, REINFORCE, baselines, and GRPO/DrGRPO in turn. All four compute log-probabilities of the tokens the agent itself generated; they differ only in the weight each token gets.
Akari Asai's L10 splits deep research agents into three problems: evaluation has to cover four gaps (search difficulty, domain expertise, long-form answer quality, citation support); training runs mid-training → SFT → RL, with DR Tulu's evolving rubrics as the reward for long-form reports; retrieval should let the retriever see the agent's reasoning, which gets AgentIR-4B to 68% on BrowseComp-Plus with Tongyi-DR.