🌏 中文版
Video status: recordings require login or course authorization. Recording source
How the model reads a history, learns from targets, and chooses its next token are different questions. Attention mixes information; the training objective updates parameters; decoding selects output under a fixed model. Keeping them distinct prevents confusing a sampling change with a model update.
Choose a lesson
- Causal attention: work through its mechanism, example, and answer criteria.
- Training and decoding: work through its mechanism, example, and answer criteria.
- Modern Transformers: work through its mechanism, example, and answer criteria.
The course map lists the core path. This page is a topic guide, not an additional lecture. Choose the question you need to answer and follow its lesson.
A worked comparison
Teaching example: fix next-token probabilities at A=0.6, B=0.3, C=0.1. Greedy picks A. Top-p=0.8 retains A and B, renormalizes to 2/3 and 1/3, then samples. Temperature 0.5 squares and renormalizes the original probabilities, yielding approximately 0.783, 0.196, and 0.022. None of these operations performs backpropagation or changes model weights. Attention acts earlier, when constructing representations of the history.
Check your understanding
What does top-p=0.5 retain, and does it change the training loss?
Answer and criterion
Only A is retained because its mass already exceeds the threshold. The decoding distribution becomes A=1. With model and training data fixed, this sampling setting does not alter the original training loss, though generated outputs and their evaluation can change.
How to use this map
Before reading the answer, identify the inputs, what changes, and the outputs. Then complete the linked lesson’s variation. If two mechanisms seem identical, trace where their conditions or data paths diverge. This page checks relationships and boundaries; it does not replace the derivations, implementations, or evaluations in the individual lessons.
The source edition is Spring 2026. Teaching examples are not official assessments or reconstructed classroom speech. Recordings remain subject to course authorization.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, the anonymous reader returned an unloaded folder and a sign-in prompt; it did not provide playable lecture content. This establishes an access limit, not the absence of recordings. This guide uses public written materials and provides no reconstructed classroom speech or original-audio transcript.
Update Log
-
2026-10-11: Rebuilt the teaching scope, worked example, and self-check; retained the original URL and publication date.
-
2026-10-10: Added explicit video status and checked recording sources and access notes.
-
2026-10-10: Rechecked video status. The login wall is confirmed (anonymous Panopto folder lists no videos and prompts sign-in); the official pages link no public YouTube version, so the status is unchanged.
References
- CMU 10-423/623/723 homepage (Spring 2026)
- Schedule: L2 and L3 dates and readings
- Lecture 2 slides: Transformer Language Models
- Lecture 3 slides: Learning Large Language Models
- Lecture 3 slides (inked)
- Goodfellow, Bengio & Courville, Deep Learning, Chapter 10: L2 assigns 10.10–10.12 (LSTMs and other gated RNNs)
- Vaswani et al. (2017), Attention Is All You Need
- Alammar (2018), The Illustrated Transformer
- Graves (2014), Generating Sequences With Recurrent Neural Networks
- Mikolov et al. (2010), Recurrent neural network based language model
- Radford et al. (2018), Improving Language Understanding by Generative Pre-Training (GPT-1)
- Radford et al. (2019), Language Models are Unsupervised Multitask Learners (GPT-2)
- Practice Exam and Solutions