LLM reasoning

1. Summary

Meta: Source type: local timestamped SRT plus source video; title: CME295 Lecture 6 - LLM Reasoning; course: Stanford CME295 Large Language Models; language: English; duration: 01:47:07; transcript type: timestamped SRT with repeated caption overlap cleaned for measurement; generated date: 2026-09-10.

Core: Reasoning models extend vanilla LLMs by spending inference-time compute on a reasoning chain, then train that behavior with verifiable rewards, GRPO-style RL, and pipeline choices that control quality, length, readability, and distillation.

Abstract: The lecture connects Lecture 5 preference tuning to modern reasoning models. It starts from RLHF, PPO, rewards, KL penalties, and the LLM-as-policy view, then defines reasoning as multi-step problem solving rather than factual recall. It explains why chain-of-thought helps: decomposition turns a hard prompt into tractable subproblems, and longer generation gives the model more forward-pass compute. The lecture then covers how reasoning is evaluated through coding and math benchmarks, pass@K, temperature, and consensus. The training core is RL with verifiable rewards: formatting rewards enforce think/answer structure, correctness rewards come from test cases or ground-truth math answers, and GRPO replaces PPO's value model with group-relative advantages. The final third uses DeepSeek R1-Zero and R1 as the concrete recipe: RL-only proves the signal is powerful, cold-start SFT improves readability, further RL and SFT mix reasoning and non-reasoning data, and distillation transfers reasoning traces from a large teacher into smaller models.

Concept: Reasoning model - an LLM that generates intermediate reasoning tokens before its final answer, using extra inference compute to solve multi-step tasks.

Concept: Verifiable reward - a reward that can be checked without a learned reward model, such as code passing tests or a math answer matching ground truth.

Concept: GRPO - group relative policy optimization; an RL method that compares multiple completions for the same prompt to estimate advantage without a value function.

Concept: Pass@K - a benchmark estimate of the probability that at least one of K sampled attempts succeeds.

Concept: Reasoning distillation - training a smaller student on complete teacher-generated reasoning traces and answers, rather than only matching token distributions.

TakeHome: Reasoning is not only better prompting; it is a training-and-inference regime that buys performance with generated reasoning tokens.

TakeHome: GRPO matters because it makes verifiable-reward RL cheaper than PPO by removing the jointly trained value model.

TakeHome: DeepSeek R1 shows the practical recipe: prove RL can elicit reasoning, then add cold-start data, readability rewards, mixed SFT/RL, and distillation to make the model useful.

2. Knowledge Tree

LLM reasoning/
├── Starting point: aligned vanilla LLMs/
│   ├── pretraining: text and code structure
│   ├── SFT: task behavior
│   ├── RLHF: preference reward and PPO
│   └── vanilla limitations: reasoning, freshness, action, evaluation
├── Reasoning behavior/
│   ├── multi-step problem solving
│   ├── chain-of-thought as decomposition
│   ├── reasoning chain plus answer
│   ├── thinking UI and hidden raw traces
│   └── reasoning tokens as paid inference compute
├── Evaluation/
│   ├── coding benchmarks: tests and bug fixes
│   ├── math benchmarks: parseable final answers
│   ├── pass@K estimator
│   ├── temperature-diversity tradeoff
│   └── consensus@K and self-consistency
├── Training from verifiable rewards/
│   ├── think-token formatting reward
│   ├── correctness reward from tests or answers
│   ├── budget control and continuous thoughts
│   └── RL as the scalable alternative to hand-written SFT traces
├── GRPO mechanics/
│   ├── multiple completions per prompt
│   ├── group-normalized advantage
│   ├── no value function
│   ├── KL control and clipping
│   └── length-bias fixes: DAPO, Dr. GRPO, asymmetric epsilons
└── DeepSeek case study/
    ├── R1-Zero: RL-only proof of concept
    ├── R1: cold-start SFT, RL, rejection-sampled SFT, final RL
    ├── helpfulness, harmlessness, and language consistency
    └── distillation into smaller reasoning models

3. Feynman Questions

Question 1. Why does the lecture treat LLM reasoning as multi-step problem solving rather than as ordinary factual question answering?

Question 2. How does chain-of-thought help a next-token model solve problems that probably never appeared verbatim in pretraining?

Question 3. What does pass@K estimate, and why is the estimator written as one minus the probability that all selected attempts fail?

Question 4. Why does reasoning training prefer verifiable rewards over hand-written SFT traces when starting from scratch?

Question 5. What is the core difference between PPO and GRPO in how they compute advantage?

Question 6. Why can the original GRPO length normalization push the model toward longer bad outputs?

Question 7. What did DeepSeek R1-Zero prove, and why was it still not enough as a final product recipe?

Question 8. How does DeepSeek R1 combine cold-start data, RL, SFT, rejection sampling, and final alignment into a usable reasoning model?

Question 9. Why is reasoning distillation different from the earlier distillation method used for next-token prediction?

4. Narrative Study Notes

001 · From preference tuning to reasoning

00:00:05-00:08:50

The lecture opens by placing LLM reasoning after the previous training stack. Pretraining teaches a model the structure of text and code, but the resulting model mainly autocompletes sequences. SFT then teaches a useful task behavior, such as answering user queries. Preference tuning adds a third stage: humans compare completions, a reward captures what they prefer, and RL updates the model toward outputs that score better. The recap matters because reasoning models reuse the same RL machinery; the new part is not the existence of rewards, but the way the reward is tied to multi-step problem solving. In the LLM-as-RL analogy, the model is the agent, the current token context is the state, the next token is the action, the token distribution is the policy, and the reward comes from a completion-level judgment. The lecture's first anchor is that reasoning training is built on preference-tuning machinery, but redirects the reward from human taste toward solved problems.

Concept: RLHF bridge - preference tuning gives the lecture its language of policies, rewards, advantages, KL control, and constrained updates. Recap slide comparing traditional RL and LLM RLHF

Mechanism: Preference tuning supplies the reusable control pattern: increase probability of rewarded behavior while KL or clipping keeps the model close enough to its earlier competence. This matters for reasoning because the later stages also need a way to explore better outputs without destroying the language and instruction-following behavior already learned.

Notes: PPO has two protective forces: maximize advantage from rewards and penalize updates that move too far from the old or base/SFT model.

StudyCue: When watching this recap, track the continuity from Lecture 5: reward, advantage, policy ratio, clipping, and KL are not side topics; they are the vocabulary of reasoning-model training.

Anchor: Reasoning RL is easiest to understand when treated as preference tuning with a different reward source: solved problems replace preferred completions.

LinkBack: Transcript 00:00:05-00:08:50.

002 · Why vanilla LLMs are not enough

00:08:50-00:12:50

After the RLHF recap, the instructor defines the target by contrast. A vanilla LLM receives a prompt and directly produces an answer. That behavior is powerful for writing, code generation, debugging, and other pattern-rich tasks, but it has several weaknesses. First, sophisticated math or coding problems may require structured intermediate reasoning, and next-token prediction alone does not guarantee the model will maintain a correct multi-step plan. Second, pretraining data is static, so factual knowledge is bounded by a cutoff date unless another system retrieves fresh context. Third, a vanilla chat model talks but does not act in the world. Fourth, free-form outputs are hard to evaluate with older NLP metrics such as BLEU or ROUGE. The lecture will later leave freshness, actions, and evaluation to future lectures; here it isolates reasoning as the weakness to repair. The useful contrast is simple: vanilla LLMs can sound plausible, but reasoning models are trained to spend computation before committing to an answer.

Concept: Vanilla LLM - the baseline prompt-to-answer model whose output is not explicitly structured as reasoning plus final response. Slide listing practical considerations and caveats of vanilla LLMs

Mechanism: The weakness list separates several failure modes that often get conflated: reasoning failure is about deriving an answer, freshness failure is about missing context, action failure is about agency, and evaluation failure is about measuring open-ended text. The lecture improves only the first one today.

Notes: The lecture explicitly narrows today's topic to reasoning; up-to-date knowledge, tool action, and broad evaluation come later.

StudyCue: Do not define reasoning by product hype; define it by the task pressure: a prompt becomes a reasoning prompt when the answer depends on intermediate transformations rather than direct recall.

Anchor: The lecture isolates reasoning by separating it from freshness, tools, and evaluation, so the training story stays focused on derivation.

LinkBack: Transcript 00:08:50-00:12:50.

003 · What counts as reasoning

00:12:50-00:22:18

The instructor gives a working definition because the field does not have a single agreed one: reasoning is the ability to solve a problem, especially a math or coding problem, through a multi-step process. A factual question such as the course code is not the central case; a problem that requires decomposing the task, using context, and deriving an answer is. Chain-of-thought supplies the historical link. Earlier prompting methods placed examples in context that showed the model how to think in steps before answering. The reasoning-model program scales that idea: instead of only prompting a model to imitate a few examples, train it so its natural output contains a reasoning chain followed by an answer. This helps for two reasons. Decomposition maps a hard unseen problem to smaller patterns the model may have seen during training, and each extra generated token runs another forward pass, increasing test-time compute. Reasoning models turn chain-of-thought from a prompting trick into an output format and training target: prompt in, reasoning chain out, answer last.

Concept: Chain-of-thought - explicit intermediate steps that make the model solve smaller subproblems before producing the final answer. Terminology slide defining reasoning as solving a problem

Mechanism: Chain-of-thought helps because it changes both the representation and the compute budget: the model writes intermediate states that make the next token easier and spends more forward passes before the final answer. This is why reasoning chains are not merely explanations after the fact.

Notes: The lecture treats retrieval context as an extension rather than the foundation of reasoning; fetching missing information belongs to the next lecture.

StudyCue: The bear-age example is intentionally simple; its role is to mark the difference between knowing a fact, having context, and using steps to transform context into an answer.

Anchor: Chain-of-thought matters because it changes the model output from answer-only text into a computation trace plus final answer.

LinkBack: Transcript 00:12:50-00:22:18.

004 · Reasoning models in products and APIs

00:22:18-00:30:59

The lecture then turns from definition to observability. Reasoning models became visible in late 2024, starting with OpenAI's o1 preview, followed by Gemini thinking models, DeepSeek R1, and reasoning additions from other labs. In user interfaces, the signal is often a word such as "thinking" or a selectable thinking level. The instructor stresses that the user-visible thought summary is not necessarily the raw reasoning chain. A provider may hide the raw chain because it is not human-readable, because users do not want pages of internal tokens, or because raw traces can become training data for imitators. The same hidden reasoning matters economically: API pricing often counts reasoning tokens as output tokens even when the full trace is not shown. Reasoning tokens are both a capability mechanism and a cost center, so the practical goal is maximum reasoning benefit per generated reasoning token.

Concept: Thought summary - a product-facing summary of hidden reasoning, not necessarily the literal token sequence used internally. Slide showing how to spot a reasoning model from thinking indicators

Mechanism: Product UIs expose only a controlled view of reasoning because raw traces create three pressures: readability for users, cost in output tokens, and model-copying risk if traces become training targets. The same hidden work is therefore both useful and commercially sensitive.

Notes: The hidden-chain issue later motivates distillation: if traces are available, another model may learn to mimic reasoning behavior from them.

StudyCue: Remember the pricing implication: a reasoning model may charge for internal reasoning tokens even when the user sees only a short thought summary and final answer.

Anchor: Hidden reasoning creates a design triangle: user-facing summaries, provider-side cost, and protection of traces that can train imitators.

LinkBack: Transcript 00:22:18-00:30:59.

005 · Benchmarks, pass@K, and sampling temperature

00:30:59-00:48:15

Reasoning models need benchmarks whose answers can be checked. Coding benchmarks ask the model to solve a problem or fix a bug, then verify the solution with test cases; HumanEval, Codeforces, and SWE-bench appear as examples. Math benchmarks ask the model to output a parseable final answer, often boxed or otherwise constrained, then compare it with ground truth; AIME and GSM8K are examples. The metric pass@K estimates the probability that at least one of K generated attempts succeeds. The instructor derives it by sampling n attempts, counting c correct ones, and computing one minus the probability that all K selected attempts come from the n-c failures: pass@K = 1 - C(n-c, K) / C(n, K). The metric matters because some settings can afford multiple attempts and a verifier. Temperature controls the diversity-quality tradeoff: too low gives similar answers, too high damages quality, and an intermediate value can improve pass@K. Consensus@K adds another idea: choose the answer that appears most often, connecting to self-consistency. Pass@K turns extra sampling into a measurable compute-for-success tradeoff: generate several attempts, verify them, and ask whether at least one works.

Concept: Pass@K estimator - 1 - C(n-c, K) / C(n, K), derived by subtracting the all-fail event from one. Blackboard derivation of pass at K

Mechanism: Pass@K is a test-time scaling metric because it assumes you can sample multiple attempts and verify them: more samples can improve success only when diversity creates genuinely different candidates and a checker can identify the correct one. Temperature controls the first condition; tests or ground truth control the second.

Notes: Temperature is part of the benchmark result, not a minor implementation detail, because it changes the diversity of sampled attempts.

StudyCue: The complement trick is the key derivation move: count the easier event that all selected attempts fail, then subtract it from one to get at least one success.

Anchor: Pass@K is meaningful only when sampling creates diverse attempts and the task provides a reliable verifier for success.

LinkBack: Transcript 00:30:59-00:48:15.

006 · Training with verifiable rewards

00:48:15-00:53:20

Once the lecture has defined reasoning and how to measure it, it asks how to train it. SFT looks tempting, but it needs high-quality reasoning traces. Long, clean, human-written chains are expensive, and human reasoning may not match the model's most effective internal style. The alternative is RL, because math and coding tasks naturally produce verifiable rewards. The training reward can combine a formatting check and a correctness check. The formatting reward asks whether the model used the expected thinking markers, such as start/end think tokens. The correctness reward asks whether the generated answer solves the problem: code passes tests, or a math answer matches the reference answer. This means the training signal does not require a learned reward model. The scalable training move is to reward the model for producing a reasoning container and for ending with a verifiably correct answer, not for imitating hand-written reasoning prose.

Concept: Formatting reward - a simple rule-based reward for emitting the expected think/answer structure. Slide on developing test-time scaling with natural verifiable rewards

Mechanism: Verifiable rewards shift supervision from path-writing to outcome-checking: the trainer does not need to know the perfect reasoning trace if it can verify the final answer and enforce the trace container. This is the central reason RL becomes attractive for reasoning.

Notes: Reasoning RL can start from scratch when trace data is weak because correctness supplies the reward even when the reasoning path is self-generated.

StudyCue: Keep formatting reward and correctness reward distinct: formatting asks whether the model thought in the required channel; correctness asks whether the resulting answer solved the problem.

Anchor: Verifiable-reward RL replaces expensive trace annotation with cheap outcome checks, which is why it scales for math and code.

LinkBack: Transcript 00:48:15-00:53:20.

007 · Compute budgets and controlling thinking

00:53:20-00:58:30

After showing that verifiable-reward RL can improve AIME-style benchmark performance, the lecture introduces the practical control problem: not every prompt deserves the same amount of reasoning. A simple question should not consume a long chain, while a hard proof or coding task may need more. One strategy is a lightweight classifier that predicts whether the prompt is high-thinking or low-thinking. Another constraint is context length: reasoning tokens occupy the same finite context window that the answer and input must share. Budget forcing offers a direct interface: insert tokens that make the model continue thinking, such as "wait," or force a stop with a phrase like "time is up, now my answer is." The instructor also mentions continuous thoughts, where thinking happens in hidden representations rather than ordinary language tokens, as an active research direction. Reasoning quality is not only a training problem; deployment must allocate, extend, stop, or compress thinking so compute follows task difficulty.

Concept: Budget forcing - inference-time intervention that pushes the model to think longer or stop sooner by inserting control tokens or phrases. Slide listing ideas to control thinking at inference time

Mechanism: Compute budgets make reasoning conditional rather than universal: the system should spend long chains on prompts where extra search changes the answer, and stop early when more tokens mainly add cost. This converts reasoning from a static model property into an inference-time allocation problem.

Notes: The open problem is economic as much as algorithmic: reasoning helps only if its marginal benefit justifies token cost and latency.

StudyCue: Connect this section to pass@K: both ask how much extra test-time compute should be spent before the expected accuracy gain stops paying for itself.

Anchor: Thinking budgets turn reasoning from a fixed model behavior into an allocation policy over latency, cost, and expected accuracy.

LinkBack: Transcript 00:53:20-00:58:30.

008 · GRPO: group-relative advantage without a value model

00:58:30-01:06:00

GRPO, or group relative policy optimization, is introduced as the go-to RL algorithm for reasoning training. Like PPO, it tries to maximize advantage while preventing the policy from moving too far from an old or reference model. The key difference is advantage estimation. PPO uses completion rewards plus a jointly trained value function to estimate expected future reward; that value model is expensive and awkward in the language setting. GRPO instead samples multiple completions for the same prompt, scores each one, and compares each completion's reward to the group's reward statistics. In the lecture's example, a completion's advantage is its reward minus the group mean, divided by the group standard deviation. This makes the same raw reward context-sensitive: solving a hard prompt among many failures should be treated differently from solving an easy prompt where everyone succeeds. GRPO replaces PPO's learned value baseline with a group baseline: sample several completions, score them, and train on how each answer compares to its siblings.

Concept: Group-relative advantage - a normalized comparison of one completion's reward against other completions for the same prompt. Slide introducing GRPO as a common RL algorithm for reasoning

Mechanism: GRPO's group baseline makes reward relative to prompt difficulty: a correct completion on a hard prompt receives a stronger advantage because the other sampled completions reveal that success was rare. This is the intuition behind sampling several completions for the same prompt.

Notes: In reasoning tasks the reward model can disappear entirely because correctness is checked by a verifier rather than learned from preferences.

StudyCue: If PPO asks "how good was this action relative to a learned value estimate," GRPO asks "how good was this completion relative to its sibling completions for the same prompt?"

Anchor: GRPO makes advantage prompt-relative by comparing completions inside the same group instead of predicting value with another model.

LinkBack: Transcript 00:58:30-01:06:00.

009 · GRPO versus PPO in the loss

01:06:00-01:15:10

The instructor then compares GRPO and PPO diagrammatically and mathematically. In GRPO, a query passes through the policy model, the model generates G completions, each completion receives a reward, the group rewards define advantages, and those advantages update the policy with KL control against a reference model. PPO follows a different path: one prompt-completion pair receives a reward, a per-token value model estimates future return, generalized advantage estimation combines reward and value, and both the policy and value function are trained. Both losses use a ratio between the current policy probability and the old policy probability, and both use clipping to avoid overly large updates. A technical difference is that GRPO writes KL divergence explicitly in the objective, while PPO implementations often fold KL into the reward/advantage calculation. The instructor lowers the cognitive load with a practical rule: PPO is heavily used for preference tuning, while GRPO is common in reasoning training. For this lecture, the important distinction is operational: PPO trains a policy plus value function, while GRPO trains only the policy and gets advantage from grouped completions.

Concept: KL control - a penalty or constraint that prevents the updated model from drifting too far from a reference distribution. Slide comparing GRPO and PPO computation graphs

Mechanism: The comparison is not merely mathematical notation; it changes the model stack: PPO requires maintaining and training an extra value model, while GRPO spends extra sampling compute to avoid that model. For LLM reasoning, that trade is attractive because verifiers can score complete answers directly.

Notes: The lecture treats the full formulas as digestible later; the stable takeaway is the value-function removal and group-relative baseline.

StudyCue: When revising the formulas, start from shared structure first: both methods use policy ratios and clipping, then differ in where the advantage and KL terms come from.

Anchor: The practical GRPO/PPO contrast is model economy: grouped samples replace the value function as the source of advantage.

LinkBack: Transcript 01:06:00-01:15:10.

010 · The length pathology and GRPO fixes

01:15:10-01:29:40

The lecture then explains why reasoning outputs tend to grow. During RL, average response length increases as the reasoning chain becomes more elaborate, and early length growth correlates with better performance. The problem appears when performance stabilizes but output length keeps increasing. The instructor traces one hypothesized cause to the GRPO objective's 1 / |O_i| factor. Because this factor depends on output length, a token in a short output receives a larger weight than a token in a long output. With positive advantage, short correct outputs get upweighted more; with negative advantage, short bad outputs get downweighted more. That creates a perverse asymmetry: a short bad output is punished more than a long bad output, so the model can drift toward longer incorrect reasoning. Recent fixes attack the normalization. DAPO equalizes token-level contributions with a common normalization factor; Dr. GRPO removes the factor altogether. Other modifications adjust the reward standard deviation term for problem difficulty or use asymmetric clipping epsilons so low-probability tokens can grow without allowing high-probability tokens to collapse. The pathology is not "long reasoning is bad"; the pathology is that the objective can reward length after accuracy has stopped improving.

Concept: Length normalization bias - the 1 / |O_i| term can make short bad outputs more strongly penalized than long bad outputs. Slide on increasing output length phenomenon in GRPO

Mechanism: The length issue is a reward-design problem, not a moral preference for short answers: if the objective punishes short failures harder than long failures, the model can learn length as a way to reduce penalty rather than improve reasoning. Fixes therefore target token-level weighting and clipping geometry.

Notes: This section links back to API pricing: controlling reasoning length matters because extra hidden reasoning tokens cost money and provider compute.

StudyCue: Separate two curves in your head: useful length growth tracks accuracy early, while pathological length growth continues after benchmark performance has plateaued.

Anchor: Length control is a reward-shaping problem because the objective can accidentally make verbosity cheaper than concise failure.

LinkBack: Transcript 01:15:10-01:29:40.

011 · DeepSeek R1-Zero as proof of concept

01:29:40-01:35:50

Shervin takes over by showing how the pieces combine in the DeepSeek papers. The traditional pipeline starts with a pre-trained base model, then applies SFT and RL alignment. DeepSeek R1-Zero deliberately tests a more radical idea: start from a pre-trained DeepSeek V3-style base model and apply GRPO directly on reasoning data, without prior SFT alignment for reasoning. The reward contains answer correctness and formatting, including the think/answer template. The result is striking: benchmark accuracy improves under RL alone, proving that verifiable rewards can elicit reasoning behavior from a base model. But the method also exposes quality issues. Without strong supervised examples, the model may mix languages, produce syntax problems, or create reasoning traces that are hard to read. R1-Zero proves that RL alone can discover useful reasoning behavior, but it also shows why raw reward optimization is not the same as a deployable assistant.

Concept: R1-Zero - DeepSeek's RL-only reasoning proof of concept, trained from a base model with verifiable and formatting rewards. Slide describing DeepSeek R1-Zero training recipe

Mechanism: R1-Zero is valuable because it isolates the RL signal: starting from a base model and adding only verifiable/format rewards shows that reasoning behavior can emerge without a large hand-written reasoning dataset. The observed language-mixing and syntax issues then identify what the next pipeline stages must repair.

Notes: The lecture's logic is experimental: R1-Zero answers "can RL elicit reasoning?" before R1 answers "how do we make that reasoning usable?"

StudyCue: Treat R1-Zero as an ablation in story form: remove supervised reasoning data, keep RL rewards, and see both the surprising capability gain and the usability defects.

Anchor: R1-Zero proves the reward signal is powerful, while its readability failures prove that reward alone does not define product quality.

LinkBack: Transcript 01:29:40-01:35:50.

012 · DeepSeek R1 as a full recipe

01:35:50-01:43:24

R1 keeps the same base-model starting point but adds stages to repair R1-Zero's weaknesses. First, DeepSeek creates cold-start data by taking reasoning chains and using humans to rewrite them for formatting, language consistency, and readability, then uses those prompt-response pairs for SFT. This gives the model a small but clean behavioral anchor before heavy RL. Next comes RL with a richer reward: verifiable answer correctness, formatting, and a language-consistency reward based on the ratio of target-language tokens in the chain. After that, the pipeline performs larger-scale SFT with both reasoning and non-reasoning data. Non-reasoning examples come from the general V3 assistant data, while reasoning examples are generated with rejection sampling: prompts are answered by the current model, judged or filtered, and only high-quality answers are kept. The final RL stage mixes reasoning rewards with helpfulness and harmlessness rewards; harmlessness applies to the whole generated sequence, including the hidden thinking section. R1's lesson is that a strong reasoning model is a pipeline, not a single trick: clean seed traces, verifiable RL, filtered SFT, and final alignment each solve a different failure mode.

Concept: Cold-start SFT - a small supervised stage that anchors formatting and readability before large-scale reasoning RL. Slide on DeepSeek R1 large-scale SFT stage with reasoning and non-reasoning data

Mechanism: R1's stages form a repair sequence: cold-start SFT cleans the format, RL raises reasoning accuracy, rejection-sampled SFT broadens high-quality behavior, and final RL restores assistant alignment. Each stage has a reason; none is decorative.

Notes: R1 uses both reasoning and non-reasoning data because a deployed assistant must solve hard tasks without losing ordinary helpfulness.

StudyCue: The 3:1 reasoning-to-non-reasoning mix is a useful anchor: the model is pushed toward reasoning capability while still being trained on general assistant behavior.

Anchor: R1 is a staged repair pipeline: each added dataset or reward targets a concrete failure in raw reasoning optimization.

LinkBack: Transcript 01:35:50-01:43:24.

013 · Distilling reasoning into smaller models

01:43:24-01:47:07

The lecture ends by asking what to do if one cannot run a huge reasoning model. Earlier distillation matched a student's next-token probability distribution to a teacher's distribution over a fixed dataset. Reasoning distillation changes the object being transferred. The teacher model, here R1, generates full sample responses, including thinking tokens and final answers. Then a smaller student is trained with SFT to reproduce the entire generated sequence. This is still distillation because a large model transfers capability to a smaller one, but it is not only probability-distribution matching at each token; it is supervised imitation of complete reasoning traces. The reported result is that distilled smaller models can be competitive with closed-source smaller reasoning alternatives, and direct RL from scratch at small scale may be less efficient than learning from a strong teacher's traces. For smaller models, reasoning distillation turns a large model's hidden work into supervised data, making imitation more efficient than rediscovering the behavior from scratch.

Concept: Trace distillation - using teacher-generated reasoning chains and answers as the supervised target for a smaller student model. Slide comparing earlier distillation with reasoning trace distillation

Mechanism: Reasoning distillation works because the expensive teacher converts hard prompts into supervised traces: the student no longer has to discover the reasoning policy through RL; it can imitate the teacher's complete think-plus-answer sequence. This explains why smaller distilled models can outperform small models trained from scratch with the same RL recipe.

Notes: The closing link to earlier lectures is important: distillation is the same compression goal, but the target has shifted from probability distributions to full reasoning sequences.

StudyCue: Earlier distillation preserved soft probabilities; this version preserves behavior: the teacher supplies the trajectory, and the student learns to reproduce the trajectory token by token.

Anchor: Reasoning distillation makes smaller models efficient by converting a large model's search behavior into supervised imitation data.

LinkBack: Transcript 01:43:24-01:47:07.

5. Suggested Answers

Question 1. Why does the lecture treat LLM reasoning as multi-step problem solving rather than as ordinary factual question answering?

Answer 1. A factual question can be answered by recall, but the lecture's reasoning cases require decomposition, intermediate steps, and a derived answer; reasoning is the model's ability to solve a problem through a multi-step process, not merely retrieve a known fact.

Question 2. How does chain-of-thought help a next-token model solve problems that probably never appeared verbatim in pretraining?

Answer 2. Chain-of-thought breaks the hard unseen prompt into smaller subproblems whose patterns are more likely to resemble training examples, and it gives the model more forward-pass compute; the reasoning chain converts one hard prediction into a sequence of easier predictions.

Question 3. What does pass@K estimate, and why is the estimator written as one minus the probability that all selected attempts fail?

Answer 3. Pass@K estimates the probability that at least one of K sampled attempts succeeds. The complement is easier to count: select K attempts from n total attempts and ask whether all K come from the n-c failures; therefore pass@K = 1 - C(n-c, K) / C(n, K).

Question 4. Why does reasoning training prefer verifiable rewards over hand-written SFT traces when starting from scratch?

Answer 4. Long human-written reasoning traces are expensive and may not match the model's best reasoning style, while math and coding tasks already provide checkable outcomes; verifiable rewards let RL train reasoning from correctness and formatting without first collecting massive human trace data.

Question 5. What is the core difference between PPO and GRPO in how they compute advantage?

Answer 5. PPO estimates advantage from rewards plus a jointly trained value function, while GRPO samples multiple completions for the same prompt and compares each reward to the group's rewards; GRPO removes the value model and uses the group as the baseline.

Question 6. Why can the original GRPO length normalization push the model toward longer bad outputs?

Answer 6. The 1 / |O_i| factor makes tokens in short outputs carry more weight than tokens in long outputs, so short bad outputs are penalized more strongly than long bad outputs; the objective can prefer a longer wrong answer over a shorter wrong answer.

Question 7. What did DeepSeek R1-Zero prove, and why was it still not enough as a final product recipe?

Answer 7. R1-Zero showed that a base model trained with GRPO and verifiable rewards can acquire reasoning benchmark performance without prior reasoning SFT, but its chains could mix languages or have syntax/readability problems; RL-only reasoning works as a proof of concept, not as a complete assistant recipe.

Question 8. How does DeepSeek R1 combine cold-start data, RL, SFT, rejection sampling, and final alignment into a usable reasoning model?

Answer 8. R1 first anchors the model with cleaned cold-start reasoning traces, then applies reasoning RL with correctness, formatting, and language consistency rewards, then uses rejection-sampled reasoning data plus general assistant data for larger SFT, and finally runs mixed RL for reasoning, helpfulness, and harmlessness; each stage repairs a specific weakness left by the previous stage.

Question 9. Why is reasoning distillation different from the earlier distillation method used for next-token prediction?

Answer 9. Earlier distillation fit the student's token distribution to the teacher's distribution on fixed data, while reasoning distillation has the teacher generate full reasoning traces and answers that the student imitates by SFT; the distilled object is the whole reasoning sequence, not only a next-token probability distribution.