🌏 中文版
Video status: recordings require login or course authorization. Recording source
A prompt with one demonstrated answer is not the same data as a prompt with a preferred and rejected response. SFT learns demonstrated tokens. A typical RLHF pipeline learns reward and then updates a constrained policy. DPO forms a loss directly from preference pairs and reference-relative ratios.
Choose a lesson
- SFT and RLHF: work through its mechanism, example, and answer criteria.
- DPO preference objective: work through its mechanism, example, and answer criteria.
The course map lists the core path. This page is a topic guide, not an additional lecture. Choose the question you need to answer and follow its lesson.
A worked comparison
Teaching DPO example: the preferred response has policy probability 0.4 and reference probability 0.2; the rejected response has 0.2 under both. The difference between reference-relative log ratios is log(2). With beta=1, sigmoid(log2)=2/3 and negative-log preference loss is approximately 0.405. That does not mean answer accuracy is 2/3: it is the objective value for this pair. SFT on the demonstrated preferred response does not use these two reference-relative ratios.
Check your understanding
What is the loss if policy and reference match on both responses?
Answer and criterion
The difference is zero, sigmoid is 0.5, and loss is log2, approximately 0.693. This is a neutral objective value for one pair, not a claim about all evaluation tasks.
How to use this map
Before reading the answer, identify the inputs, what changes, and the outputs. Then complete the linked lesson’s variation. If two mechanisms seem identical, trace where their conditions or data paths diverge. This page checks relationships and boundaries; it does not replace the derivations, implementations, or evaluations in the individual lessons.
The source edition is Spring 2026. Teaching examples are not official assessments or reconstructed classroom speech. Recordings remain subject to course authorization.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, the anonymous reader returned an unloaded folder and a sign-in prompt; it did not provide playable lecture content. This establishes an access limit, not the absence of recordings. This guide uses public written materials and provides no reconstructed classroom speech or original-audio transcript.
Update Log
-
2026-10-11: Rebuilt the teaching scope, worked example, and self-check; retained the original URL and publication date.
-
2026-10-10: Added explicit video status and checked recording sources and access notes.
-
2026-10-10: Rechecked video status. The login wall is confirmed (anonymous Panopto folder lists no videos and prompts sign-in); the official pages link no public YouTube version, so the status is unchanged.
References
- CMU 10-423/623/723 Generative AI (Spring 2026) course home page
- Course schedule: L11 and L12 dates, Quiz 3
- Lecture 11 slides: In-Context Learning / Instruction Fine-tuning / RLHF
- Lecture 12 slides: DPO / Latent Diffusion Models (inked version)
- HW3 handout (hw3.zip): structure and points of the DPO section
- Practice Exam (Spring 2026)
- Ouyang et al. 2022: Training language models to follow instructions with human feedback (InstructGPT)
- Bai et al. 2022: Training a Helpful and Harmless Assistant with RLHF
- Rafailov et al. 2023: Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Schulman et al. 2017: Proximal Policy Optimization Algorithms
- Williams 1992: Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Wei et al. 2021: Finetuned Language Models Are Zero-Shot Learners (Flan)
- Longpre et al. 2023: The Flan Collection: the instruction-dataset comparison figure in the slides
- Xu et al. 2023: MultiInstruct
- databricks/databricks-dolly-15k