The Knobs You Turn When an LLM Generates, from Temperature and Top-p to Chain of Thought | Stanford CME295 Lecture 3

🌏 中文版

Video status: Videos included. Source details

For the 2026 edition, use the Lecture 2 recording and slides. It covers topics from both 2025 Lecture 2 and Lecture 3; this guide keeps its original 2025 lecture number. The “What changed in 2026” section explains the comparison.

This post covers Lecture 3, "Large Language Models," of the 2025 edition of Stanford's CME295 (October 10, 2025). The main source is the 125-slide deck; the recording runs 1 hour 48 minutes. Everything here comes from the text and figures on the slides. Anything said only out loud in class is not included.

If you have called any LLM API, you have probably seen temperature and top_p in the parameter list, and maybe an option for forcing JSON output. Where each of those fields intervenes in the model's work is the core of this lecture.

The slides are organized into five parts: LLM overview, MoE-based LLMs, Response generation, Prompting strategies, and Inference optimizations. This post centers on the middle three, which are the knobs you can turn while the model is generating. MoE gets the intuition only, and the final part on inference speedups gets a map, with links to fuller posts on this site for the details.

What counts as an LLM

The slides open with a definition: a language model is a statistical or machine learning model that "assigns probabilities to sequences of tokens." The "Large" rests on three things:

• Model size: billions of parameters or more
• Training data: hundreds of billions of tokens or more
• Compute: in the slides' words, "a lot of GPUs"

Architecturally, an LLM in this course means a decoder-only Transformer, with GPT, LLaMA, Gemma, DeepSeek, Mistral, and Qwen listed as examples. The previous lecture (Lecture 2) split Transformers into encoder-decoder (T5), encoder-only (BERT), and decoder-only (GPT). This lecture only deals with the third kind.

MoE: a huge model, but each token only walks a small part of it

The starting point on the slides is one observation: "Not all weights are useful in the forward pass." For any particular input, a giant model may not need every weight. So why not run only part of it each time?

The recipe is to split one big network into n "experts," E1 through En, and put a gating network G in front to decide which to use:

• Dense MoE: every expert runs, and the output is a weighted average of all of them. Nothing is saved.
• Sparse MoE: G picks a few experts via top-k, only those run, and their outputs are averaged. This is the version that saves compute, from Shazeer et al. (2017).

Inside a Transformer, the part that gets replaced is each layer's feed-forward network (FFNN). The slides flag it in bold: "Routing done for each token!" In a single sentence, "teddy" and "reading" can go to different experts.

The best-known problem in training MoE is routing collapse. The gating network finds one expert slightly better and keeps choosing it. The other experts get no training signal and fall further behind, which is a vicious circle. The slides cite the fix from Switch Transformers: an auxiliary loss that forces the other experts to be "part of the game" too.

loss = α · N · Σ_{i=1..N} f_i · P_i

What do the experts actually learn? The slides show a figure from the Mixtral paper that colors each token of a Python snippet by the expert it was routed to. The paper's own conclusion is that experts do not split by topic, but the router does "exhibit some structured syntactic behavior." Python's self, for instance, often goes to the same expert. So the division of labor looks syntactic, not like a "math expert" and a "code expert."

The full set of MoE trade-offs (load-balancing variants, communication cost, why nearly every 2026 frontier model uses it) is covered in two posts on this site: CS336 Lecture 4 and Why MoE Wins.

Knob 1: how the next token is chosen

At every step an LLM outputs a probability distribution over the vocabulary. You pick one token from it, append it, and feed the sequence back in. The slides walk through three ideas in turn:

Two common ways of truncating the candidates for sampling:

• top-k: sample only among the k most probable tokens; the slide example uses k = 4
• top-p: sort tokens by probability, take the smallest set whose cumulative probability is ≥ p, and sample from it; the slide example uses p = 90%. The method comes from Holtzman et al.'s nucleus sampling paper

The difference shows up when the distribution is very peaked or very flat. When the model is confident, the top two or three tokens already cover 90%, and top-p keeps only those. When it is unsure, top-p lets in many more candidates. top-k keeps exactly k no matter what the distribution looks like.

Knob 2: temperature reshapes the distribution

Where do the probabilities come from? The decoder's final linear layer produces a score for every token, then softmax turns the scores into probabilities. Temperature sits inside that softmax: each score is divided by T first.

P_adj(w_{t+1} = w_i | C) = exp(x_i / T) / Σ_{j=1..n} exp(x_j / T)

The slides show the effect with two bar charts. With a small T, almost all the probability lands on one token (kind in the example), which is close to greedy. With a large T, every candidate is about equally likely and the output turns random. So temperature controls how willing the model is to pick an unlikely word, and top-k/top-p control how unlikely a word can be before it is ruled out entirely. The two are usually used together.

The slides add a suggested reading here: Thinking Machines' Defeating Nondeterminism in LLM Inference. It tackles why the same prompt, run twice on an inference service with temperature set to 0, can still give different results.

Knob 3: guided decoding bans invalid tokens outright

If you want JSON and only write "answer in JSON" in the prompt, the model may still add a friendly opening line or drop a bracket. The slide example asks for a description of "my 33-year-old teddy bear who likes reading" as {"first_name": "teddy", "last_name": "bear", "age": 33, "hobby": "reading"}.

Guided decoding only allows "valid" next tokens at each step. The first token can only be {, then only a key string, and after a key only :. Every other token is made impossible, so sampling can never land on it. This is the idea behind the "structured output" features in vendor APIs.

flowchart LR
  L["Linear layer scores<br/>(one logit per token)"] --> T["Divide by temperature T"]
  T --> S["softmax<br/>probability distribution"]
  S --> G{"Format constraint?"}
  G -->|yes| M["guided decoding<br/>remove invalid tokens"]
  G -->|no| F
  M --> F["top-k / top-p<br/>cut the tail"]
  F --> P["sample<br/>(near-greedy as T→0)"]
  P --> N["append the token<br/>feed back, next step"]
  N --> L

Knob 4: context length, and fitting is not the same as reading well

The slides give three synonyms for how much input a model reads at once: context length, context size, and window size. They give no concrete numbers, only that the order of magnitude depends on input type and model. Beside that, in red: "Beware of "context rot"!", citing Chroma's Context Rot study, which found that performance drops as input grows even when the task itself does not get harder.

This site has a post that collects long-context failures and how different tools respond to them: Seven Answers to a Full Context Window.

Knob 5: prompting changes the output without touching weights

The first four knobs act on decoding. This part changes the input instead. The slides first break a prompt into four parts, using a bedtime story for a tired teddy bear:

Then three techniques, each with its cost spelled out on the slides:

In-context learning (ICL). From the GPT-3 paper. Zero-shot means asking without examples, so performance depends entirely on the model. Few-shot means putting a few input/output examples in the prompt, which typically works better. The cost is the effort of writing examples and a longer prompt, which raises compute cost and latency.

Chain of thought (CoT). From Wei et al. (2022), built on the idea that "explaining reasoning helps in improving performance." The slide example first shows a worked question: "The bear was born in 2020. It is therefore 4." Then it asks how old the bear will be next year, and the model follows the pattern, reasoning before answering: "It will be one year older than its age this year, which was 4. Hence, it will be 5." The upside is interpretability, since you can see how the model got there. The downside is that the extra tokens add cost and latency.

Self-consistency. From Wang et al. (2022): sample several reasoning paths for the same question and aggregate the answers. In the slide example, two of three paths get 5 and one wrongly gets 4, so the majority answer is 5. It ties straight back to Knob 1: you only get different paths if you sample, not if you decode greedily. The cost scales with the number of paths, and the slides frame it as a "trade-off between performance and added cost."

The last part: a map for making generation faster

This part is not in the 2025 syllabus topic list, but it takes up roughly the last 40 slides, and Part III of the midterm asks four questions about it. The slides split inference speedups into exact optimizations that do not change the output and approximations, and cover six techniques:

If Q_i(token) >= P_i(token): accept
Otherwise: accept with probability Q_i(token) / P_i(token),
           reject with probability 1 - Q_i(token) / P_i(token)
On rejection: resample one token from [Q_i - P_i]+ (normalized), then stop this round

MQA/GQA already came up in Lecture 2; here they return from the angle of the KV cache being too large. For the full picture of inference optimization, see CS336 Lecture 10 and the vLLM deep dive that covers PagedAttention.

Back to the models you use

Mapping this lecture onto everyday API use:

• You need stable, reproducible output (field extraction, classification, code): lower the temperature. When the format must be fixed, use the API's structured output feature. It is more reliable than writing "please output JSON" in the prompt, because it blocks invalid tokens at decoding time.
• You want varied output (brainstorming, copywriting): raise the temperature and pair it with top-p to cut off the very-low-probability tail, so you don't sample something completely off-topic.
• The model gets reasoning questions wrong: try few-shot plus chain of thought first. When accuracy matters more than cost, add self-consistency and take the majority over several samples.
• You stuffed in a long document and answers got worse: that may be context rot. Instead of pasting the whole document, retrieve the relevant passages first, which is where Lecture 7 on RAG starts.

MoE and inference speedups are things the model provider does for you. What you notice is the result: faster responses and lower inference cost.

What changed in 2026

The lecture that absorbs this one is 2026 Lecture 2 (held October 2), and the 2026 syllabus already lists its slides and recording (the second video in the source section below). The comparison uses the syllabus topic lists and the transcript checks recorded on October 10; it is not a slide-by-slide comparison of the 2026 deck:

• This lecture is folded into 2026 Lecture 2, "Large Language Models." Its topic list is Transformer model families, LLM definition and architecture, Mixture of experts, MHA/MQA/GQA, RoPE and variants, context length, temperature, and sampling strategies. In effect, 2025 Lecture 2 and the first half of 2025 Lecture 3 become one lecture.
• Prompting, in-context learning, chain of thought, and self-consistency are not listed in the syllabus as separate items, but they still get a short closing segment in 2026 Lecture 2. The syllabus topic lists do not name them; I read the transcript of the Lecture 2 recording, and its last minutes cover the context window and context rot, prompting and in-context learning without changing weights (noting that few-shot is used less and less), and chain of thought and self-consistency, the latter two only touched on and deferred to the later reasoning lectures.
• Inference speedups move into a whole new lecture. 2026 Lecture 5, "LLM systems," lists inference optimizations, KV caching, speculative decoding, and Flash Attention, so the last part of this 2025 lecture will likely be expanded there; see the preview in order 10 of this series.
• Guided decoding does not appear in the 2026 syllabus either, but the Lecture 2 transcript does cover it: restricting the tokens allowed at each step by the target syntax so that output such as JSON is guaranteed valid. So it was folded into the decoding section, not cut. KV cache, speculative decoding, and Flash Attention do not appear in this video's transcript, which is consistent with the guess that inference speedups moved to Lecture 5.

Self-check

These questions are adapted from Part III of the 2025 midterm. Answers are in the solutions PDF:

1. In a sparse MoE, how does routing decide which experts each token uses? (Q2)
2. How does speculative decoding speed up generation? What do the draft and target models each do? (Q4)
3. How is the candidate set in top-p sampling chosen, and how does it differ from top-k? (Q7)
4. Raising the temperature during decoding makes the distribution sharper or flatter? (Q8)
5. What is routing collapse? Name one standard mitigation. (Q9)
6. Compare greedy/beam search with top-k/top-p sampling on diversity, quality, and compute. (Q10)

Going deeper

• Full MoE and attention-variant trade-offs: CS336 Lecture 4: Attention Has Alternatives, and MoE Does Not Scale for Free
• Why frontier models moved to MoE: Why MoE Wins
• Another take on decoding strategies: CS224N Lecture 12: Decoding, DeepSeek-R1, and Reasoning Training
• Where in-context learning comes from: CS224N Lecture 7: Pretraining, Subwords, and In-Context Learning
• Inference optimization: CS336 Lecture 10
• Next in this series: Lecture 4: LLM training; training chain of thought into the model itself comes in Lecture 6: LLM reasoning

Course video sources

The first video comes from Stanford Online’s official CME295 Autumn 2025 playlist (Lecture 3); the second is the Autumn 2026 recording listed for Lecture 2, "Large Language Models", on the official 2026 syllabus (https://cme295.stanford.edu/syllabus/). Lecture titles and video IDs were checked live on 2026-10-10 and match.

Original videos: Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 3 - Tranformers & Large Language Models, Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

Content check: verified against the video transcript (2026-10-10): For both videos I sampled the beginning, middle and end of the transcript, searched keywords, and compared against the chapter list in the video description (not a word-by-word comparison). Q5baLehv5So is Autumn 2025 Lecture 3 (2025-10-10, 1:48:45, matching the post); its chapters are MoE, decoding (greedy, beam, sampling, temperature), guided decoding, prompting, in-context learning, CoT and self-consistency, KV cache, and PagedAttention/MLA, consistent with the post's "knob" sections. GaIeu3npx04 is Autumn 2026 Lecture 2, Large Language Models (1:43:14); its transcript really is the LLM lecture (MoE, RoPE, sliding windows, GQA, temperature, guided decoding, context rot, prompting). Note that the chapter list in its page description is copied from Lecture 1 and does not match the content, so I went by the transcript. Found and fixed: the post said prompting, in-context learning, CoT and self-consistency disappeared from the 2026 syllabus and that guided decoding could not be placed, but the 2026 Lecture 2 transcript briefly covers the first four at its end and also covers guided decoding.

Course and recording entries:

• Official course / lecture source
• CME295 Autumn 2025 playlist (Stanford Online, 9 videos)

Checked: 2026-10-10.

Update Log

• 2026-10-11: Added direct links to the 2026 Lecture 2 recording and slides and clarified how it maps to the 2025 guides.
• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-10-10: Rechecked video status. Checked live against the 2025 playlist and the official 2026 syllabus; the 2025 Lecture 3 and 2026 Lecture 2 video IDs match, and the 2026 Lecture 2 recording was added.
• 2026-10-10: Fixed an outdated recording statement. The 2026 Lecture 2 slides and recording are now posted, so the "only Lecture 1 is out" wording was rewritten.
• 2026-10-10: Checked the video content against its transcript. Found that the 2026 Lecture 2 recording does cover prompting, ICL, CoT and self-consistency (at the end) and guided decoding, and corrected the post's "disappeared / cannot tell" statements; everything else matches.

References

• 2026 Lecture 2 recording

• 2026 Lecture 2 slides (PDF)

• CME 295 2025 syllabus

• CME 295 2026 syllabus

• 2025 Lecture 3 slides (PDF)

• 2025 Lecture 3 recording

• 2025 midterm / solutions

• Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)

• Fedus et al., Switch Transformers (2021)

• Jiang et al., Mixtral of Experts (2024)

• Holtzman et al., The Curious Case of Neural Text Degeneration (2019)

• He, Defeating Nondeterminism in LLM Inference (Thinking Machines, 2025)

• Hong et al., Context Rot (Chroma, 2025)

• Brown et al., Language Models are Few-Shot Learners (2020)

• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)

• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022)

• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)

• Ainslie et al., GQA (2023)

• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023)

• DeepSeek-AI, DeepSeek-V2 (2024)

• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023)

• Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction (2024)

• Reading Stanford CME295 (series overview)