Reasoning: From CoT and Test-Time Scaling to RL for Reasoning | NTU Yun-Nung Chen Applied Deep Learning Lecture 12

🌏 中文版

Video status: Videos included. Source details

This post is based on the Lecture 12 videos (12.1–12.5) in the playlist of NTU Applied Deep Learning (ADL), Fall 2025 (114-1, 2025/09/01–12/15), taught by Yun-Nung (Vivian) Chen. It is Lecture 12 in the Reading NTU Yun-Nung Chen Applied Deep Learning series. The previous post, Language Agents, treated reasoning as one of three key concepts for agents. This lecture pulls it out on its own: how does a model learn to think before it answers, and to think correctly?

This lecture has no public slides, and the videos have no captions. I transcribed the audio of the five videos and wrote this post from the transcripts, so the content is what the instructor said in Mandarin. Numbers marked "on the slide" are ones I read from the slides visible in the video frames; other numbers attributed to her are ones she spoke aloud. Examples marked "made up for this post" are not from the lecture.

After this lecture you should be able to do five things: say how test-time scaling differs from learning to reason; compute pass@k; pick an answer once each with majority voting, answer confidence, an ORM, and a PRM; explain why training data should not keep only perfect reasoning; and state the difference between R1-Zero and R1.

Why everyone is suddenly talking about reasoning

The instructor first separates two words that Mandarin often translates the same way. Inference is the point in time when a trained model is being used. Reasoning is logical derivation of the form "because A and because B, therefore C." This lecture is about the second.

Then the motivation. Each generation of GPT, Claude, and Gemini beats the last, but the gains keep shrinking. One explanation she relays: earlier generations could still add more training data, and now the amount of data used to train models is about to catch up with the total stock of public human text. Pre-training is running out of new data to add.

So where does the next generation's improvement come from? OpenAI's o1 pointed in one direction. From the user's side it is simple: you ask a question, and it thinks for a while before replying. The thinking does produce tokens. The version released at the time hid them, and API users were still billed for them, so longer thinking costs more.

She uses a chart to make the point (she mentions that OpenAI published an article explaining o1; the chart on the slide is titled "o1 AIME accuracy" with no printed citation, and matches the one in that article). The vertical axis is accuracy on AIME competition math. One panel's horizontal axis is train-time compute and the other's is test-time compute, and accuracy rises in both. The meaning: the same model, with no further training, still improves if it is allowed to think longer or more times at inference. Few groups can train a large model, but anyone can try a different inference strategy, which is why much academic work moved to this side.

There are now many such models; she names o1, o3, the Thinking versions of Gemini, DeepSeek's R1 series, and some versions of Claude. Their output starts with thinking wrapped in a <think> block and ends with the answer. That thinking shows the strategies people use on hard problems: verifying, trying another route, and breaking a big problem into smaller ones.

A note on the o1 date: the video says December 2024, which is when the full version of o1 was released. The article with that chart, "Learning to reason with LLMs," and o1-preview came earlier, on 2024-09-12.

Short CoT: give worked solutions, or just say "step by step"

Thinking more at inference starts with Chain-of-Thought in 2022.

A plain few-shot prompt gives a few question-answer pairs and then asks a new question. Math questions often go wrong, and her explanation is that a language model continues text, and the probability of continuing a question directly with the right number is hard to estimate. CoT changes each example to question, worked solution, answer. An example might spell out "there were 5 balls, 2 cans of 3 were bought, so 6 more, 5 + 6 = 11." The model imitates the example and writes intermediate steps first. Each step is a short continuation, and reaching the right answer becomes much easier. She says that with this kind of prompt alone, language models at the time caught up on math problems with models built specifically for them. She says the author, when visiting Taiwan, described the idea as similar to teaching his children. Her own gloss: telling a child "you got it wrong" does not help next time; a worked solution does.

Later work found the examples could be dropped. Adding "Let's think step by step" is enough for a model that follows instructions well to write steps before answering, and it does better than asking for the answer directly. This is zero-shot CoT. It is now a common experimental baseline, because few-shot prompts require explaining how many examples were used, where they came from, and how they were chosen.

She sums these methods up in one framing that the rest of the lecture uses: a piece of reasoning is a path. A question has many reasoning paths, some ending at the right answer and some at a wrong one. Every method aims to raise the chance of reaching a correct path.

From short CoT to long CoT

The few-step kind above is short CoT. Today's reasoning models think for much longer. Interfaces usually collapse the thinking, and some models show only a summary. She introduces a survey of long CoT (the slide credits Chen et al., 2025, Towards Reasoning Era) and lists three features that long CoT adds:

All three can appear inside one long path in a single response.

Spending more at inference (1): answer several times, then pick one

The most direct method is parallel sampling: have the model answer the same question k times, with a higher temperature so the answers spread out (temperature is covered in Lecture 9).

The metric for "did we have a chance of reaching a correct path" is pass@k: the question counts as solved if any of the k answers is right. Larger k gives higher pass@k (the pass@k chart on the slide is credited to Brown et al., 2024, Large Language Monkeys). She adds two cautions:

• The model's own ability matters. A weak model may get nothing right in 10 samples and only start producing correct answers after hundreds or thousands.
• pass@k is an upper bound, not the score you get. In practice you do not know which answer is right, so you still have to choose.

Three common ways to choose (the slide credits one paper for each: Wang et al., 2022 for majority voting; Wang and Zhou, 2024 for answer confidence; Cobbe et al., 2021 for the ORM):

1. Majority voting. Pick the answer that appears most often. This is also called self-consistency: if the model gives the same answer across many attempts, it is more confident in that answer.
2. Answer confidence. Look at the probability the model assigned when decoding the answer token itself; higher probability is more trustworthy. This is the answer token, not the probability of the whole output. The slide's example is "I have 3 apples, my dad has 2 more apples than me, how many apples do we have in total": of five paths, three say 5 and two say 8, but confidence on 8 is clearly higher, so 8 is chosen. On the slide the five paths come from taking each of the top five first tokens and continuing greedily. The GSM8K results on the slide: greedy decoding 44.0%; decoding 10 paths and ranking by the whole output's log-prob 37.0%, or 51.0% with length normalization; ranking by answer confidence 72.0%.
3. Outcome reward model (ORM). Train a separate model as a verifier. It takes the question and a full response and outputs how likely the response is to be correct; pick the highest score. This is also called best-of-n. The verifier is trained on a different dataset that has ground-truth answers: let an existing model solve those problems, mark which responses are right, and that is the training data. The ORM can itself be a language model. She stresses that how well this selection works depends largely on how well the ORM is trained.

When comparing methods, the horizontal axis should be total tokens generated, not number of attempts, because responses differ in length. On such a chart, majority voting sits between pass@1 and pass@k, and ORM sits a little above majority voting.

The video does not mention the following numbers; I added them. OpenAI's article has a set that lines up with these three tiers: on the 2024 AIME, o1 averaged 74% with one sample per problem, 83% with consensus among 64 samples, and 93% when re-ranking 1000 samples with a learned scoring function.

A worked example (made up for this post)

I made up the numbers below so each selection method can be computed once. Suppose the correct answer to a question is 8. The model answers 5 times, and each response has three reasoning steps:

• pass@5: D and E are right, so it is 1. This only says the correct answer was sampled.
• Majority voting: 5 gets 3 votes and 8 gets 2. It picks 5 and is wrong.
• Answer confidence: here I average the probabilities for each distinct answer. For 5 the average is (0.40 + 0.45 + 0.35) ÷ 3 = 0.40; for 8 it is (0.90 + 0.85) ÷ 2 = 0.875. It picks 8 and is right. The video does not say how to aggregate; averaging is my assumption.
• ORM (best-of-n): the highest score is D's 0.70. It picks 8 and is right.
• PRM: average the step scores to score each response. A is 0.47, B is 0.50, C is 0.43, D is 0.87, E is 0.83. It picks D and is right.

The point of the example: the correct answer was sampled, and majority voting still chose wrong, because wrong paths can simply be more numerous. Answer confidence and reward models use information beyond the vote count. One caveat: a poorly trained verifier makes best-of-n choose wrong too, so it is not guaranteed to beat majority voting.

One more calculation for the upper bound. If the model answers independently and is right with probability 0.3 each time, then pass@5 = 1 − 0.7⁵ ≈ 0.83. A single attempt succeeds 30% of the time, yet at least one of five succeeds 83% of the time. Going from 0.3 to 0.83 is the upper bound that more samples provide; how much of it you actually get depends on the selection method.

Spending more at inference (2): score as you go and cut bad paths early

An ORM can only score a response once it is complete. If a path has already gone wrong halfway, the remaining tokens are wasted. Hence the process reward model (PRM), which scores each step of the reasoning.

She presents PRM use in two stages:

1. First change only the scoring. Every path still runs to the end, and the step scores are aggregated (most simply averaged) into a score for the whole response. That alone does somewhat better than ORM: the result on the slide is 72.4% for ORM and 78.2% for PRM (69.6% for majority voting), a gap of almost 6 percentage points.
2. Then add search. With a score at every step, low-scoring paths can be stopped midway and the saved compute given to paths that still look promising. Beam search and Monte Carlo style search both fit. With beam search, for example, you start with four paths, stop the low-scoring ones, and extend only the good ones; a budget that would have taken every path only to the second level can now take the remaining paths one level further. Some methods also look one step ahead and come back to revise if it looks bad.

Back to the table above: paths A, B, and C score 0.2, 0.3, and 0.2 at step two. Stopping them there saves three step-three generations, which could instead expand more branches from paths like D and E.

Parallel or sequential?

In parallel sampling each attempt is independent. The alternative is sequential revision: after the first version is written, hand it back to the model, ask it to check what is wrong, and write another version, repeatedly. Each version is still a full response, so an ORM can score each one.

An analysis she cites (credited on the slide to Snell et al., 2024) compares the two. With the same number of generations, on harder problems such as math, sequential revision beats parallel sampling once the number of generations grows, whether the final choice uses majority voting or reward-model weighting. The only difference between them is whether the model can see the previous version. Even a wrong previous version carries useful information. Her one-line takeaway: thinking deeper beats thinking many times.

Parallel and sequential can also be mixed, and she says that then comes close to the search methods above.

Following that direction, she describes a piece of work from her own lab. The model that solves the problem is called the policy model, and thinking alone it tends to get stuck in one line of thought. So a second nudging agent is added. Like a tutor, it watches each step, analyzes it, judges it positive or negative, and gives a short suggestion when it is negative. Her positioning: it does what a PRM does, and adds a suggestion on top of the score. The suggestion is written into the same response, so it reads as the model telling itself "no, this part should be changed to…". The response gets longer and explores more directions, and she says the result is higher than best-of-n with a PRM. The slide credits it to Yen-Ting Lin's 2025 PhD thesis.

The other side: update the weights so the model learns to reason

Everything above leaves the model untouched. The second half of the lecture, learning to reason, fine-tunes it.

What a piece of reasoning looks like

Her example is a permutation question she asked Gemini. The slide shows it in full: arrange the seven characters of 「庭院深深深幾許」 in a line; how many arrangements have exactly two of the character 「深」 (shēn) adjacent? The model's thinking went like this:

• It confirmed the goal and noted there are three 「深」.
• Strategy one: glue two adjacent 「深」 into one unit, count 6! = 720 arrangements, subtract the 240 where all three are together, and get 480.
• Strategy two: verify with the gap method. Partway through it noticed the gaps were set up wrong, wrote "Let's refine the slotting," fixed it, and also got 480.
• Only after both methods agreed did it write up the final answer.

The seven characters are three 「深」 plus four distinct ones. The numbers for both methods are on the slide; the explanations in parentheses are mine:

• Gluing: treat two 「深」 as a block. With the remaining 「深」 and four distinct characters there are 6 units, giving 6! = 720. Arrangements with all three 「深」 adjacent number 5! = 120, and each is counted twice by the gluing method (block then single, and single then block), which the slide writes as 120 + 120 = 240; subtracting gives 480.
• Gap method: arrange the four distinct characters in 4! = 24 ways, creating 5 gaps. The block and the single 「深」 go into different gaps, in P(5, 2) = 5 × 4 = 20 ways. 24 × 20 = 480.

She points out one more thing: she asked in Chinese, the model answered in Chinese, and its thinking was in English. A reasoning path can run in any language, and which one is more likely to go right depends on how familiar the model is with that language. The model does not always think in English either; sometimes it thinks in Chinese.

Where the data comes from

To teach reasoning with supervised learning, the training data needs the intermediate thinking as well as inputs and outputs. There are three sources:

1. Written by people. Ask olympiad gold medalists to write out how they solve problems, or ask doctors to write how they reason toward a diagnosis. Quality is high, but collecting it at scale is hard.
2. Generated by a model, then filtered. Have a model solve problems from a dataset with known answers and keep the paths whose answers are right. More strictly, also check with a PRM and keep only paths where every step scores high.
3. Generated by a stronger model (distillation). Covered below.

The second approach has a trap. Keeping only paths where every step is correct shows the model nothing but perfect solutions. The most valuable part of the Gemini example was that it went wrong midway, noticed, and corrected itself. Training only on perfect data means the model never learns to notice and fix its own mistakes, and both the amount and the diversity of data shrink.

So imperfect paths are worth keeping. Data produced with the nudging agent is one source: it contains right steps, wrong steps, and revisions made after a nudge. After SFT (supervised fine-tuning, see Lecture 7) on such long data, a model that could only do short CoT produces a token-length distribution shifted clearly toward longer outputs, and its performance rises again. On the slide, short responses average about 656 tokens (median 524) and long ones about 2,743 (median 1,497).

Keeping the wrong data, and still telling the model what is good

She connects back to DPO and KTO from Lecture 7, which train a model on which output people prefer. Here, the reward model's score serves as the preference.

Step-KTO adds a layer: besides outcome-level feedback on the whole response, it uses process rewards to give feedback on each step. The model learns which steps in a long path are right and which are wrong but later corrected, so it does not treat the wrong steps as something to imitate. The table on the slide compares three models (Llama-3.1-8B, Llama-3.1-70B, Llama-3.3-70B) on MATH-500, AMC23, and AIME24: in most columns Step-KTO scores above KTO with outcome feedback only, and above the rejection approach that simply discards samples with wrong answers. For example, pass@1 on MATH-500 with Llama-3.1-8B is 53.8 for rejection, 60.6 for KTO, and 63.2 for Step-KTO.

Distillation: let a strong teacher write the solutions

Since strong reasoning models already exist, they can generate reasoning paths to train smaller models. Her analogy: the teacher prepares worked solutions, and the student drills through them. Reasoning data from DeepSeek-R1 distilled into models such as Qwen and Llama, which could not reason before, gives them decent results on problems that need reasoning.

The limit is worth knowing: the student only imitates the teacher. On a hard problem unlike anything seen before, the teacher model has a better chance of exploring a new path, and the student lacks that exploratory ability. Many use cases never need reasoning that complex, and a student model is enough.

One trade-off when using reasoning models

How long a reasoning model thinks can be controlled by a budget, or written into the instruction, such as "I'm in a hurry, I need the answer now" or "think carefully, take your time." On hard problems, spending more tokens on reasoning helps. The cost is higher latency and higher price. She says balancing latency against performance is quite hard. Some questions need little thought, and even with a large budget the ideal model should answer briefly.

Using reinforcement learning to let reasoning emerge

The last video covers another route: give no examples of thinking and let the model develop reasoning behavior by itself. She says this route works better and is also harder. This is the approach of DeepSeek-R1.

R1-Zero is direct. Give the model problems, let it try anything, and place no limits on how it thinks. At the start it only needs a light trigger, such as asking it to go step by step or raising the temperature, so that it has a chance to produce different results. The problems have known answers, and the model is told only whether its final answer is right. That signal is the reward for reinforcement learning. The training curve on the slide: pass@1 on AIME starts at about 0.15, climbs slowly through training to around 0.7, and approaches o1, drawn as the reference line (pass@1 about 0.74); the other curve is consensus over 16 samples, which ends above 0.8, and the chart's second reference line is o1 with consensus over 64 samples (about 0.83). She says the start is "about 0.1%," most likely a slip for 0.1 (a bit over ten percent); the paper gives 15.6% rising to 71.0%. During training, very colloquial lines appeared in the model's thinking without being taught, such as this one on the slide: "Wait, wait. Wait. That's an aha moment I can flag here."

R1-Zero's weakness is that its reasoning is messy and hard to read, sometimes switching between Chinese and English, because training placed no constraint on what it wrote. The released R1 therefore adds steps:

1. Filter the reasoning produced by R1-Zero to keep the readable part, add a small amount of human-annotated data, and run supervised training on the base model first. She describes this as handling cold start: the model first learns a decent way to write reasoning.
2. Run RL from that model. Besides answer correctness, the reward adds constraints such as language coherence.
3. Use the resulting model to generate reasoning data for inputs from more domains, filter again, retrain the base model, and run RL once more (the slide labels this last step with safety and helpfulness; she does not elaborate).

So the released R1 is not a pure RL model; R1-Zero is. She says the details in between need not be memorized, and one idea is enough: get reasoning ability from pure RL first, then filter the data repeatedly to raise quality and readability.

Why reasoning research is almost all math and code

She closes with an explanation. Many tasks need reasoning, such as commonsense inference or writing a report, and those cases may be more complex than math. Research concentrates on math and code because their answers are easy to verify: math has a correct answer, and running a program shows whether the result is right. For other tasks it is hard to evaluate objectively whether the reasoning was correct, so progress in reasoning is often tied to evaluation methods.

My own summary: this also shows the premise shared by several methods in this lecture. Training an ORM, filtering reasoning paths, and the reward for R1-Zero all require knowing whether the answer is right. Majority voting, answer confidence, and sequential revision do not need ground-truth answers.

Common misreadings

• "The model thinks longer because it was retrained." Not necessarily. Test-time scaling methods leave the weights untouched.
• "High pass@k means the system is accurate." pass@k only says the correct answer was sampled. Being able to select it is a separate matter.
• "Reasoning training data should be as clean as possible." Filtering out every wrong step means the model cannot learn to notice and fix errors.
• "A distilled small model reasons as well as its teacher." It has learned the kinds of paths the teacher walked, and its ability to explore on new hard problems is weaker.

Self-check

1. The same model produces five candidates for one question and then picks one. Is that retraining?

No. The weights are not updated; this is test-time scaling. More candidates only raise the chance of sampling the correct answer, and a reliable way to select is still needed.

2. A model is right with probability 0.5 per attempt and answers independently 3 times. What is pass@3? What is the probability that majority voting is right? (Assume all wrong attempts give the same wrong answer.)

pass@3 = 1 − 0.5³ = 0.875. Majority voting needs at least 2 correct: 3 × 0.5² × 0.5 + 0.5³ = 0.375 + 0.125 = 0.5. Note that majority voting at 0.5 is no better than answering once, so voting gains nothing in this setup. The gap between 0.5 and 0.875 can only be claimed by a selection method better than majority voting.

3. A reasoning path has PRM scores 0.9, 0.1, 0.9, 0.9 and a correct final answer. When building SFT data, should it be dropped because step two scored low?

By this lecture's argument, not automatically. First check whether the path corrects itself after step two: a model going wrong and then fixing it is exactly the behavior worth teaching. A better use is to keep the path and give step-level feedback (for example with Step-KTO) so the model knows which step was good and which was not.

4. Both ORM and PRM can select an answer. Which one can stop a path early, and why?

PRM. An ORM takes the whole response as input, so it has no score until the response is finished. A PRM scores every step, so partway through it can already judge whether to continue.

5. Change one cell in the five-sample table above: sample C's answer becomes 8, and everything else stays. What does majority voting pick? What does answer confidence pick, averaging the probabilities for each distinct answer?

Majority voting: 8 has 3 votes (C, D, E) and 5 has 2, so it picks 8. Answer confidence: the average for 5 is (0.40 + 0.45) ÷ 2 = 0.425, and for 8 it is (0.35 + 0.90 + 0.85) ÷ 3 = 0.70, so it also picks 8. The two methods agree this time.

6. How does DeepSeek-R1-Zero differ from the released DeepSeek-R1, and why were the extra steps added?

R1-Zero is pure RL: the only reward is whether the answer is right, with no reasoning examples. It gains reasoning ability, but its reasoning is messy and hard to read. R1 first runs supervised training on filtered reasoning data, then RL with a language-coherence constraint added to the reward, and later retrains on generated data for another round. The extra steps make the reasoning readable and extend it to more domains.

Further reading

• CS224N Lecture 12: Decoding, DeepSeek-R1, and Reasoning Training: R1-Zero/R1, PPO, GRPO, DAPO, matching the RL route at the end of this lecture.
• CS224N Lecture 13: Speculative Decoding and Test-Time Scaling: matching the test-time scaling of the first half.
• CME295: LLM Reasoning, plus CME295: RL with LLMs on RL training.
• Let's Verify Step by Step: the representative paper on process reward models, and the one cited on the slide that introduces PRMs.
• From the same university, the Hung-yi Lee Machine Learning 2026 Spring guide: all five ADL Lecture 12 descriptions credit the slides to Prof. Lee, so his own course is a natural companion.

Series navigation: Series overview | Previous: Language Agents | Next: Conversational AI and Tool Use

Course video sources

These videos were checked on 2026-10-10 against the official course page and official YouTube playlist (lecture numbers and titles match); no timestamp is supplied.

Original videos: 12.1 What is Reasoning?, 12.2 Short CoT, 12.3 Test-Time Scaling, 12.4 Learning to Reason, 12.5 RL for Reasoning

Course and recording entries:

• Official course and recording entry

Checked: 2026-10-13.

Content check: verified against the video transcript (2026-10-13): none of the five videos has captions, so this guide was written from speech-recognition transcripts of the audio of 12.1–12.5. The transcripts may mishear some proper nouns; for several numbers I captured video frames and checked the slides shown in them (the Gemini example, the answer-confidence table, the PRM versus ORM table, the token-length distribution, the Step-KTO table, the R1-Zero training curve, and the R1 pipeline diagram); other numbers follow what was spoken. The five videos were uploaded on 2025-11-20 with a description dated 2025/11/17 that credits the slides to Prof. Hung-yi Lee; the course page lists Reasoning under 12/01.

Update Log

• 2026-10-10: Added explicit video status and checked recording sources and access notes.
• 2026-10-10: Rechecked video status. The embedded videos match the lectures on the official course page and playlist.
• 2026-10-11: Added an invented conceptual example and comprehension check.
• 2026-10-13: Moved the verification history out of the body and switched cross-references within the series from post order to lecture numbers.
• 2026-10-13: Rewrote the whole guide from the spoken content of the five videos. The earlier version only ordered the video titles and had no lecture content; this version adds the mechanism for each part, a worked example, and a self-check with answers.
• 2026-10-13: Corrected the Gemini example from the slide frames (the character is 「深」, and the full question and both calculations are on the slide), added the citations and numbers printed on the slides (PRM versus ORM, token lengths, the three Step-KTO models), and added learning goals.

References

• NTU Applied Deep Learning Fall 2025 course page — the 12/01 row carries only the title "Reasoning"
• 2025 Fall NTU CSIE ADL playlist (lectures in Mandarin)
• Videos: 12.1 What is Reasoning?, 12.2 Short CoT, 12.3 Test-Time Scaling, 12.4 Learning to Reason, 12.5 RL for Reasoning
• Learning to reason with LLMs (OpenAI, 2024-09-12) — the o1 article; the AIME figures of 74% / 83% / 93% come from here
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv 2201.11903)
• Large Language Models are Zero-Shot Reasoners (arXiv 2205.11916) — "Let's think step by step"
• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models (arXiv 2503.09567)
• Let's Verify Step by Step (arXiv 2305.20050)
• Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback (arXiv 2501.10799)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv 2501.12948)