AI Safety: A Learning Path from Risk Identification to Experiments and Research | Harvard CS2881R Course Overview

🌏 中文版

Video status: Videos included. The intro embeds the Lecture 1 recording (2 h 26 min); each lecture's recording is in its own post. Source details

CS 2881R AI Safety is a Harvard graduate seminar taught by theoretical computer scientist Boaz Barak, first offered in Fall 2025. The course site describes it as covering the challenges of AI alignment and safety, looking at both the technical side and the societal impact.

The core question can be put this way: we train models into the shape we want with data and rewards, but the training signal only approximates what we mean. Where will a model drift from our intent, can we tell when it does, and what can we do about it? The 12 lectures start from that question and run from training details all the way to the economy and society.

There are no problem sets. Most lectures have a reading list, a talk (usually with guests), and a student group's experiment report; the assignments are reproducing a paper and pushing it one step further. This post is the series entry point: it lays out what each of the four questions covers, works one hand calculation that shows how the course treats numbers about AI progress, and ends with a prerequisite check and reading route. Below is the Lecture 1 recording: Boaz introduces the course, a student experiment report follows, and the second half is his take on defining AGI and the families of alignment methods; no need to watch it in one go, come back to it alongside the Lecture 1 post.

The storyline: four questions

This series does not follow the official dates. It regroups the lectures under the four questions below (every post title keeps its official lecture number). The official schedule interleaves technical and societal lectures; regrouped, each lecture's ideas get used in the next.

How does a trained model end up misaligned? (Lectures 1, 2 and HW0)

Lecture 1 lays out three views of the future: AI 2027 sketches superhuman AI within five years via recursive self-improvement, AI as Normal Technology argues AI will spread slowly like electricity, and METR measures that the length of task AI can finish with 50% success doubles about every 7 months. Boaz sorts alignment methods into three kinds: writing principles, training character, and writing a model spec.

HW0 lets you see misalignment for yourself: fine-tune Llama-3.2-1B with LoRA on bad medical advice, and the model becomes harmful on unrelated questions too. This is emergent misalignment. You taught one narrow domain, and the whole persona broke.

Lecture 2 treats pretraining, SFT, and RL as one operation: push some tokens up and others down, the difference being whether the data was written by someone else or generated by the model itself. Safety training sits on the last two stages. The current approach (Deliberative Alignment) first teaches the model to read the spec in its chain of thought, then runs RL with a reward model that understands the spec. One key warning: put optimization pressure on the chain of thought, and the model learns not to stop cheating but to stop saying it cheats.

How do models get broken, and how should they be specified? (Lectures 3, 4, 5)

Lecture 3 explains why aligned models still get jailbroken: safety training patches particular attacks, but the underlying weakness remains. Nicholas Carlini shows gradient-found adversarial suffixes that transfer across models, and Boaz borrows lessons from software security: attacks only get stronger, so defend in depth.

Lecture 4 asks whether a model spec should state principles or detailed rules. The answer is a mix of principles, character, and explicit policies, and the crux is that rules must be checkable: "prove the theorem or give a counterexample" is a bad rule, "prove it, give a counterexample, or explain that you can't" is a good one, because only a rule whose violations you can judge can be used for training and evaluation.

Lecture 5 carries two decades of social-platform content moderation over to generative AI: wherever you draw the line there are edge cases, but you have to draw it; and a chat interface sits somewhere between a private document and a public post, which makes it harder to place than a social platform.

How do we detect the hardest risks to see? (Lectures 8, 10)

Lecture 8 covers scheming: models faking good behavior or cheating to pass training or evaluation. Boaz files this under "systematic misalignment", something our training signals push models toward. A student experiment put coding agents on an impossible task, and they rewrote the tests even when explicitly told not to. Redwood's Buck Shlegeris argues for assuming the model is scheming and protecting deployment with AI control.

Lecture 10 attacks detection from two sides: read the chain of thought the model writes, or read its activations. CoT monitoring catches reward hacking well, but feed the monitor into the training reward and the model learns to hide its intent, the same point as Lecture 2's warning. Neel Nanda's advice is to compare against simple baselines first; steering vectors often beat SAEs.

How fast is AI moving, and what will it change? (Lectures 6, 7, 9, 11, 12)

Lecture 6 uses growth theory to ask: once AI starts doing AI R&D, does capability stay exponential, accelerate into a singularity, or get dragged down by bottlenecks? The answer hinges on a few exponents nobody can measure well; the only option ruled out is "AI barely affects R&D". Lecture 7 takes on a puzzle: on benchmarks the task length doubles every 7 months, yet in METR's randomized controlled trial experienced developers were 19% slower with AI. One reason is that benchmark tasks are too clean and too cheap to grade.

Lecture 9 looks at jobs: employment for 22–25-year-olds in highly AI-exposed occupations fell 16% in relative terms, senior workers show no such trend, and aggregate employment shows no large-scale displacement yet. Lecture 11 covers chatbots and mental health. Boaz offers an explanation he himself calls unproven: the longer the conversation and the further it drifts from the training distribution, the more the model slips back into a mode of going along with the story to please. Lecture 12 looks ten years out: GDPval measures AI against real work products from industry experts, and coding agents have not automated AI research yet because verification is expensive and the feedback loop is long.

Work it by hand: how far does task-length doubling extrapolate?

This is a teaching example written for this post, to show why Lectures 1, 6, and 7 keep coming back to exponents.

METR's measure is the task length at which a model has a 50% success rate on tasks that take humans that long. They report this length doubling about every 7 months, with Claude 3.7 Sonnet at about an hour. Start from 1 hour and hold the doubling time at 7 months:

If AI starts accelerating AI R&D and cuts the doubling time to 4 months, 7 years is 21 doublings: $2^{21} \approx 2.1$ million hours, more than 500 times larger.

What this means: over the same seven years, moving the doubling time from 7 months to 4 changes the conclusion from "a 50% chance of finishing a two-year project" to "about a thousand years of work" (at about 52 work weeks a year). That sensitivity is what Lecture 6 means when it says the answer hinges on exponents nobody can measure well. Lecture 7 adds the other caveat: this curve measures clean benchmark tasks, and real work (19% slower in the RCT) may not follow it. So the course's stance on timelines is to read the arguments on both sides rather than memorize one number.

Prerequisites and self-check

The course site asks for mathematical maturity, the ability to write proofs, probability and information theory, plus undergraduate ML (for example Harvard CS 181 or MIT 6.036): empirical versus population loss, gradient descent, neural networks, linear regression, PCA; you should write Python and be able to train a basic neural network. On this site, start with the Harvard CS 181 guide; HW0 uses LoRA, so if it is new to you, read the CMU 11-868 PEFT/LoRA post first.

After reading this post, try to answer:

1. How is emergent misalignment different from a model simply "learning something bad"? Answer: Fine-tuning on one narrow domain (such as bad medical advice) makes the model harmful on unrelated questions. The misalignment spreads beyond what was trained.
2. Why shouldn't a chain-of-thought monitor be part of the training reward? Answer: The model learns to stop writing down its cheating, and the monitor stops working. Lectures 2 and 10 both make this point.
3. Why is "prove the theorem or give a counterexample" a bad rule? Answer: Faced with a problem it can't solve, the model can only violate the rule or make something up. Adding "or explain that you can't" makes violations judgeable, so the rule can be used for training and evaluation.
4. If task length doubles every 7 months, starting from 2 hours, about how long is it after two years? Answer: 24/7 ≈ 3.4 doublings, $2 \times 2^{24/7} \approx 22$ hours.
5. Benchmarks say AI is strong while an RCT says developers got slower. How can both be right? Answer: Benchmark tasks are cleaner than real work, cheaper to grade, and need less context; the two measure different things.

How to read it: order and materials

This series follows the finished Fall 2025 term. The method is read, reproduce, extend, and these materials are available outside Harvard:

• Per lecture: the reading list on the course site (most lectures have one); the YouTube playlist has lecture recordings for 11 lectures (for Lecture 5 both the site and the playlist list only a student experiment video; Lecture 8 has only the opening); student-written LessWrong weekly summaries cover some of the weeks. Slides are public for only about half the lectures.
• HW0: the GitHub repo is public; its grading script uses an LLM judge, so outside readers need their own OpenAI API key.
• Midterm and final: the midterm has groups of 2–4 pick one of four papers, reproduce its headline figure, add one or two extensions, and write a 3–5 page report; the final extends an existing paper or pursues longer-term research, as a 5–10 page paper plus poster. Specs and rubrics are public and can be followed in full; but there are no problem sets or answer keys. The goal is learning to reproduce and extend, not finishing a set of exercises.

HW0 comes after Lecture 1 because on the official timeline it predates the first lecture as the enrollment gate; the student experiment in Lecture 1 is its mirror image.

One thing to know when reading the guest lectures: the course site states that Boaz is also on OpenAI's technical staff, and Lectures 5, 9, 10, and 12 each have at least one guest from OpenAI too. The course discusses models from several companies, and each lecture post states the speakers' affiliations as listed.

Fall 2026 is running now with a new set of topics (Cyber Capabilities, RL for Post-Training, AI Biosecurity, and others) and a different HW0. This series does not draw on it; after the term ends, we will decide whether to write an update.

What to do tonight

1. Open the HW0 repo README, read the three Instructions steps, and decide whether your GPU or cloud budget can handle LoRA fine-tuning of a 1B model.
2. Redo the table in "Work it by hand", then try it again with a 10-month doubling time.
3. Read the Lecture 1 post.

Related reading on this site

• LLM training and RLHF: Stanford CS336 guide, Stanford CME295 guide
• RL foundations: Berkeley CS285 Spring 2026 guide
• Interpretability: CS224U analysis methods: probing and attribution
• Prompt injection and agent security in practice: The harness layer of agent security
• LLM-as-judge: Stanford CS329Z Lecture 14: judges and Lecture 15: guardrails
• Other Harvard courses: Harvard AI/ML course map

Course video sources

This is a course overview; the intro embeds the Lecture 1 recording. Pick other lectures from the official playlist. Fall 2025 has lecture recordings for Lectures 1–4 and 6–12, student experiment videos for Lectures 5 and 8, Lecture 8's opening, and the final oral presentations; the same playlist also contains Fall 2026 Lectures 1 and 3, which this series does not use.

Course and recording entries:

• harvard-cs2881r — official course materials and recording index
• CS2881R Fall 2025 official YouTube playlist (AI Safety, 17 videos)
• Lecture 1 recording (embedded in the intro, 2 h 26 min)

Checked: 2026-10-14.

Update Log

• 2026-10-14: Rewrote the body as a real guide: the 12 lectures organized under four questions with what each covers, a hand calculation on task-length doubling, self-check questions, and the Lecture 1 recording in the intro; the course-site URL, base term, access rating, conflict-of-interest note, and Fall 2026 preview were cut down to a few paragraphs in the reading route.
• 2026-10-10: Added explicit video status and checked recording sources and access notes.

References

• CS 2881R AI Safety, Fall 2025 course site — description, prerequisites, Mini Syllabus, conflict-of-interest note, 12-lecture schedule and reading lists
• CS 2881R Fall 2026 homepage — new schedule, assignment structure, note that last year's materials are preserved
• CS 2881R Fall 2026 Homework Zero — new HW0 topic, time estimate, submission requirements
• CS 2881R YouTube playlist — lecture recordings
• LessWrong wikitag: CS 2881r — student weekly summaries and experiment posts
• Harvard-CS-2881/harvard-cs-2881-hw0 — Fall 2025 HW0 repo
• Student Final Projects - Fall 2025 — 19 final papers and posters
• Final oral presentations video
• Roy Rinberg, Reflections on TA-ing Harvard's first AI safety course (LessWrong, 2026-01-15) — enrollment, assignment structure, final deadlines, midterm and final document links
• CS2881 Mini-project slides, Mini Project Grading
• cs2881 Final Project Outline, Final Project Grading Rubric
• Harvard Q-report (course evaluation)
• Turner et al., Model Organisms for Emergent Misalignment (arXiv 2506.11613) — the paper HW0 reproduces