🌏 中文版
Video status: recordings require login or course authorization. Recording source
Scaling laws study model and data allocation under a budget. MoE increases expert capacity while activating selected experts. The former concerns allocation and empirical models; the latter concerns architecture and routing. Both deserve separate checks.
Choose a lesson
- Model/data allocation: work through its mechanism, example, and answer criteria.
- MoE routing: work through its mechanism, example, and answer criteria.
- Distributed training: work through its mechanism, example, and answer criteria.
The course map lists the core path. This page is a topic guide, not an additional lecture. Choose the question you need to answer and follow its lesson.
A worked comparison
Teaching budget example: use the simplified relation C∝N×D for relative comparisons. A has N=10 and D=20; B has N=20 and D=10. Both products equal 200, but equal budgets do not imply equal losses. Separately, a four-expert MoE with 100 toy parameters per expert and top-2 routing activates 200 expert parameters out of 400 total, excluding shared layers, routing, and communication. These two values of 200 have different units and cannot establish equal compute.
Check your understanding
Does changing top-2 to top-1 halve total expert capacity?
Answer and criterion
No. Total expert parameters remain 400; activated expert parameters fall to 100 in this example. Quality, load balance, and communication still require evaluation.
How to use this map
Before reading the answer, identify the inputs, what changes, and the outputs. Then complete the linked lesson’s variation. If two mechanisms seem identical, trace where their conditions or data paths diverge. This page checks relationships and boundaries; it does not replace the derivations, implementations, or evaluations in the individual lessons.
The source edition is Spring 2026. Teaching examples are not official assessments or reconstructed classroom speech. Recordings remain subject to course authorization.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, the anonymous reader returned an unloaded folder and a sign-in prompt; it did not provide playable lecture content. This establishes an access limit, not the absence of recordings. This guide uses public written materials and provides no reconstructed classroom speech or original-audio transcript.
Update Log
-
2026-10-11: Rebuilt the teaching scope, worked example, and self-check; retained the original URL and publication date.
-
2026-10-10: Added explicit video status and checked recording sources and access notes.
-
2026-10-10: Rechecked video status. The login wall is confirmed (anonymous Panopto folder lists no videos and prompts sign-in); the official pages link no public YouTube version, so the status is unchanged.
References
- CMU 10-423/623/723 Generative AI (Spring 2026) home page
- Course schedule — L15 and L16 dates, Quiz 4/5 coverage
- Lecture 15 slides: Querying Transformer + Scaling Laws
- Lecture 16 slides: Mixture of Experts, inked version
- Lecture 18 slides — exam covers L1–L15
- Spring 2026 practice exam, solutions
- Kaplan et al. 2020: Scaling Laws for Neural Language Models
- Hoffmann et al. 2022: Training Compute-Optimal Large Language Models
- Gunasekar et al. 2023: Textbooks Are All You Need
- Li et al. 2023: Textbooks Are All You Need II: phi-1.5 technical report
- Goyal et al. 2024: Scaling Laws for Data Filtering
- Cai et al. 2024: A Survey on Mixture of Experts in Large Language Models
- Muennighoff et al. 2024: OLMoE: Open Mixture-of-Experts Language Models
- Jiang et al. 2024: Mixtral of Experts
- Clark et al. 2022: Unified Scaling Laws for Routed Language Models (ICML)