🌏 中文版
Video status: recordings require login or course authorization. Recording source
A model that does not fit on one GPU has a different problem from a model that fits but decodes slowly. Distributed strategies partition data, parameters, and states. FlashAttention improves attention memory traffic; KV caching avoids recomputing historical keys and values during decoding.
Choose a lesson
- Multi-GPU training: work through its mechanism, example, and answer criteria.
- FlashAttention and efficient decoding: work through its mechanism, example, and answer criteria.
The course map lists the core path. This page is a topic guide, not an additional lecture. Choose the question you need to answer and follow its lesson.
A worked comparison
Teaching example: equal-sized local batches on two GPUs produce scalar gradients 2 and 6. Data-parallel averaging gives 4, and both replicas update with 4. Without communication they use different gradients and drift apart. During decoding, a KV cache retains historical keys and values and appends the next token. It does not average training gradients. More GPUs do not automatically remove large attention intermediates; partitioning and kernel behavior matter.
Check your understanding
Can you average 2 and 6 directly if local batch sizes differ?
Answer and criterion
If these are local mean gradients, weight them by local sample counts to obtain the global mean. This condition is unrelated to inference caching.
How to use this map
Before reading the answer, identify the inputs, what changes, and the outputs. Then complete the linked lesson’s variation. If two mechanisms seem identical, trace where their conditions or data paths diverge. This page checks relationships and boundaries; it does not replace the derivations, implementations, or evaluations in the individual lessons.
The source edition is Spring 2026. Teaching examples are not official assessments or reconstructed classroom speech. Recordings remain subject to course authorization.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, the anonymous reader returned an unloaded folder and a sign-in prompt; it did not provide playable lecture content. This establishes an access limit, not the absence of recordings. This guide uses public written materials and provides no reconstructed classroom speech or original-audio transcript.
Update Log
-
2026-10-11: Rebuilt the teaching scope, worked example, and self-check; retained the original URL and publication date.
-
2026-10-10: Added explicit video status and checked recording sources and access notes.
-
2026-10-10: Rechecked video status. The login wall is confirmed (anonymous Panopto folder lists no videos and prompts sign-in); the official pages link no public YouTube version, so the status is unchanged.
References
- CMU 10-423/623/723 Generative AI (Spring 2026) home page
- Course schedule — L17 and L18 dates, Quiz 5 coverage
- Lecture 17 slides: Distributed Training, inked version
- Lecture 18 slides: Efficient Attention (FlashAttention) & Efficient Decoding Strategies
- Shoeybi et al. 2019: Megatron-LM
- Harlap et al. 2018: PipeDream
- Narayanan et al. 2021: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- Rajbhandari et al. 2019: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Li et al. 2021: TeraPipe: Token-Level Pipeline Parallelism
- Lepikhin et al. 2020: GShard
- Fedus et al. 2021: Switch Transformers
- Milakov & Gimelshein 2018: Online normalizer calculation for softmax
- Ivanov et al. 2020: Data Movement Is All You Need
- Dao et al. 2022: FlashAttention
- Dao 2023: FlashAttention-2
- Kwon et al. 2023: Efficient Memory Management for Large Language Model Serving with PagedAttention