1. Summary
Meta: Source type: YouTube lecture; course: Stanford CME295 Large Language Models; lecture: Lecture 5 - LLM Tuning; speaker/channel: Stanford Online; language: English; duration: about 1:47:41; transcript type: timestamped SRT; generated date: 2026-09-10.
Core: The lecture explains LLM tuning as the post-SFT alignment problem: collect preference signals, learn or use rewards, optimize the model toward preferred behavior, and control the cost and instability of RL-based methods.
Abstract: Lecture 5 starts from the LLM lifecycle built in Lecture 4: pretraining gives a model broad next-token competence, and supervised fine-tuning teaches it to follow instructions, but SFT does not naturally express negative preferences or subtle behavioral comparisons. The lecture introduces preference tuning through pairwise comparisons, maps LLM generation into reinforcement learning, and builds RLHF in two stages: train a reward model from pairwise preferences, then optimize the SFT model with PPO-like objectives that increase reward while limiting policy drift. The last third motivates lighter alternatives: best-of-N reranking avoids training but moves cost to inference, while DPO turns preference pairs into a direct supervised objective that removes the explicit reward model but introduces distribution-shift trade-offs. The main study move is to see preference tuning as behavioral optimization, not fact learning: it changes which plausible completion the model chooses.
Concept: Preference tuning - post-SFT training that pushes the model toward outputs humans prefer and away from outputs humans reject.
Concept: Preference pair - a prompt with two candidate responses, usually labeled as winner and loser for the behavior dimension being optimized.
Concept: Pairwise preference data - comparison data that is easier to label than absolute scalar scores because humans decide which response is better.
Concept: Policy - in the LLM-RL mapping, the model's next-token probability distribution conditioned on the current context.
Concept: RLHF - reinforcement learning from human feedback; here, a two-stage method that trains a reward model and then uses RL to tune the policy.
Concept: Reward model - a model that maps a prompt-response pair to a scalar score intended to represent a chosen preference dimension.
Concept: Bradley-Terry model - a pairwise probability model where the chance that one response beats another depends on the sigmoid of their reward-score difference.
Concept: Reward hacking - optimizing an imperfect proxy reward so aggressively that the measured reward rises while the real objective degrades.
Concept: KL divergence - a nonnegative measure of how far one probability distribution has moved from another, used to constrain policy drift.
Concept: PPO - proximal policy optimization; an RL algorithm that limits update size while using advantages to reinforce or suppress actions.
Concept: On-policy training - training on samples generated by the current policy rather than on fixed external demonstrations.
Concept: Best-of-N - an inference-time method that samples multiple completions, scores them, and returns the highest-rated candidate.
Concept: DPO - direct preference optimization; a supervised preference-tuning method that fits pairwise preferences without an explicit reward-model training stage.
TakeHome: SFT teaches desired outputs, but preference tuning can also teach the model what not to output.
TakeHome: RLHF converts subjective human comparisons into an optimization signal, but it inherits reward-model noise, PPO instability, and multi-model training cost.
TakeHome: DPO trades maximum control for simplicity: it often gives strong preference tuning with fewer moving parts, while PPO-like RL may still win when expert tuning and compute are available.
2. Knowledge Tree
CME295 Lecture 5 - LLM Tuning/
├── Course frame and previous lecture/
│ ├── final scope: lectures 5-9
│ ├── pretraining: broad next-token competence
│ ├── SFT: instruction-following behavior
│ └── preference tuning: align outputs with human choices
├── Preference data/
│ ├── teddy-bear motivation example
│ ├── negative signal missing from SFT
│ ├── pointwise score difficulty
│ ├── pairwise winner-loser comparisons
│ ├── human labels, LLM-as-judge, rule metrics
│ └── rewrite-from-logs data construction
├── LLM reinforcement-learning mapping/
│ ├── agent: LLM
│ ├── state: current input/context
│ ├── action: next token
│ ├── policy: next-token probability distribution
│ ├── reward: preference-derived output quality
│ └── objective: align policy parameters with preferences
├── RLHF reward modeling/
│ ├── prompt-response input and scalar score output
│ ├── Bradley-Terry pairwise probability
│ ├── sigmoid of reward difference
│ ├── negative log-likelihood loss
│ ├── LLM reward head or encoder classifier
│ └── reward dimensions: useful, friendly, safe
├── RL policy optimization/
│ ├── rollout, reward score, policy update
│ ├── reward hacking and proxy failure
│ ├── KL penalty to stay near base model
│ ├── value function and advantage estimation
│ ├── PPO-Clip and PPO-KL
│ ├── model copies: current, old, reward, value, reference
│ └── variants: PPO, GRPO, other RL algorithms
├── Practical alternatives/
│ ├── on-policy versus off-policy distinction
│ ├── SFT limitation on negative examples
│ ├── best-of-N reranking
│ ├── inference cost and latency tail
│ └── training cost versus serving cost trade-off
└── Direct preference optimization/
├── PPO complaints: four models and RL complexity
├── optimal policy under reward-plus-KL objective
├── reward rewritten through policy and reference model
├── Bradley-Terry substitution
├── direct pairwise loss
├── two-model requirement: policy and reference
├── distribution shift trade-off
└── PPO versus DPO practical choice
3. Feynman Questions
Question 1. Why does Lecture 5 introduce preference tuning after SFT instead of treating SFT as enough alignment?
Question 2. What makes pairwise preference labels easier and more useful than absolute pointwise scores for LLM outputs?
Question 3. In the RL formulation for LLMs, what are the agent, state, action, policy, and reward?
Question 4. How does RLHF turn pairwise human preferences into a scalar reward model?
Question 5. Why does the Bradley-Terry formula use a sigmoid of the reward difference rather than two independent scores?
Question 6. What loss does the reward model minimize, and how does it come from maximum likelihood on winner-loser pairs?
Question 7. Why can reward hacking occur even when the optimizer is successfully increasing the reward-model score?
Question 8. Why does RLHF need a KL term or reference model instead of simply maximizing reward as hard as possible?
Question 9. What role do the value function and advantage play in PPO-style tuning?
Question 10. Why does PPO clip or penalize the ratio between the current and old policy?
Question 11. What makes PPO-style RLHF operationally expensive compared with SFT?
Question 12. Why is PPO called on-policy training, and why is SFT not on-policy in the same sense?
Question 13. Why does best-of-N avoid RL training but still create deployment cost and latency problems?
Question 14. What is the key mathematical move that lets DPO remove the explicit reward model?
Question 15. Why might PPO outperform DPO in some settings even though DPO is simpler?
Question 16. In the teddy-bear example, why is the preferred answer not mainly about factual correctness?
4. Narrative Study Notes
001 · Lecture frame: from training to tuning
00:00-05:00
The lecture opens by locating Lecture 5 in the course sequence. The midterm material is now behind the class, and the final will cover lectures 5 through 9. That logistics detail matters because this lecture starts the second half of the course: after learning what transformers are and how LLMs are trained, the course now asks how a trained model is shaped into behavior that users actually want. The speaker reviews the prior lecture's lifecycle. Pretraining begins from an initialized model and uses massive data and compute to teach language, code, and broad next-token structure. Training optimization then makes that stage feasible through GPU parallelism, data parallel methods, ZeRO variants, model parallelism, FlashAttention, mixed precision, and quantization. Supervised fine-tuning then turns the pretrained autocompleter into a model that can answer prompts for specific tasks. The key new context is that even after SFT, the model may produce completions that are fluent and task-shaped but still badly aligned with the user's preference. Lecture 5 starts where SFT stops: the model can answer, but tuning must decide which answer should be preferred.
CoreFocus: The technical pivot is from learning language and tasks to ranking behaviors under human preference.
Concept: LLM tuning - the post-pretraining and post-SFT process that changes model behavior using preference, reward, or direct comparison signals.
Mechanism: The lecture deliberately begins with the prior lifecycle because preference tuning only makes sense after pretraining and SFT have created a capable base assistant. Pretraining supplies broad token prediction, SFT supplies instruction-shaped behavior, and tuning operates on the remaining choice among plausible completions. The new object of optimization is not language competence; it is preference among already-possible behaviors.
Recap slide placing fine-tuning after pretraining
Notes: The previous lecture's LoRA/QLoRA material remains relevant because parameter-efficient adaptation can be used during tuning, but preference tuning is primarily an objective-function question, not only a parameter-count question.
LinkBack: 00:00-05:00, course logistics, recap of pretraining, SFT, and training optimizations.
002 · Preference tuning motivation: the teddy-bear example
05:00-10:00
The speaker introduces preference tuning with an SFT model that receives a friendly prompt: suggest a new activity to do with a teddy bear. The bad response is not a malformed sequence; it is an assistant-like answer that says the user should not spend much time with the teddy bear at all. This example separates grammatical competence from user-preferred behavior. A better answer acknowledges that teddy bears can be companions and suggests playful activities, so the issue is not that the model lacks English. The issue is that the response fails the preference criterion for tone, helpfulness, and user empathy. This creates a preference pair: the same prompt with one rejected response and one desired response. Preference tuning uses such pairs to align the model with human preferences. The lecture then asks why SFT is not enough. SFT requires a high-quality dataset of prompts and target responses, and building that dataset can be expensive. It also teaches the model what to imitate, but it does not naturally encode the negative signal that a specific plausible completion should be avoided. Preference tuning adds the missing comparison: for the same prompt, this response is better than that response.
CoreFocus: The teddy-bear prompt shows that alignment failures can be behavioral and tonal even when the response is syntactically valid.
Concept: Preference pair - a prompt plus two responses, one preferred and one rejected, used to teach relative behavior.
Application: In practice, this means a model can pass a surface task test and still fail a user preference test. The teddy-bear response is fluent and direct, but it violates the social expectation that an assistant should respect the user's implied affection for the object. The failure is not syntax; it is response selection under a human criterion.
Slide asking why preference tuning is needed after SFT
Notes: SFT data can include good responses, but preference data can explicitly say that another response is worse; that negative information is the new training signal.
LinkBack: 05:00-10:00, teddy-bear setup, SFT limitation, preference-pair definition.
003 · Preference data: negative signals and pair construction
10:00-15:00
The lecture clarifies a common confusion: LoRA and preference tuning are not alternatives at the same level. LoRA reduces how many parameters must be trained; preference tuning changes the objective being optimized. One can therefore use LoRA while doing preference tuning, because parameter-efficient adaptation and preference objectives solve different problems. The speaker then emphasizes the core behavioral difference from SFT. SFT teaches what the model should predict; preference tuning can teach what it should not predict. That negative signal is valuable because many alignment failures are not solved by adding one more ideal demonstration. The lecture then turns to the data problem. A prompt such as "write a poem" can produce many candidate poems. A pointwise approach asks a rater to assign each poem an absolute scalar score, but that is hard because humans are inconsistent across prompts, tasks, and rating scales. A pairwise approach compares two candidates for the same prompt and asks which one is better. This local comparison is easier, more stable, and better matched to how preference datasets are usually collected. Preference data becomes useful when it turns vague quality judgment into a concrete winner-loser comparison.
CoreFocus: LoRA changes trainable parameter count; preference tuning changes the behavioral loss.
Concept: Negative signal - training information that identifies an output the model should avoid, not merely an output it should imitate.
Mechanism: The negative signal is the reason preference tuning cannot be reduced to adding more demonstrations. If the model sees only preferred completions, the loss says imitate these targets; it does not directly lower probability on a competing rejected completion for the same prompt. Preference pairs create pressure in both directions: raise the winner and lower the loser.
Preference data slide comparing pointwise scores
Notes: Pointwise scores ask for global calibration; pairwise labels ask for a local ordering. That difference is why pairwise preference data dominates many alignment pipelines.
LinkBack: 10:00-15:00, LoRA versus objective function, pointwise scoring, negative preference signal.
004 · Pairwise labels, judges, and preference-data sources
15:00-20:00
Once the data is framed as pairs, the lecture separates the label source from the label format. The comparison can come from human raters, from an LLM-as-judge system, or from rule-based metrics such as BLEU and ROUGE, though the latter are less central for modern open-ended preference tuning. The simplest format is binary: response one is better or worse than response two. A richer format could say much better, slightly better, slightly worse, and so on, but the lecture notes that subjectivity makes these scales difficult. For many tasks, a binary pairwise preference label is the practical compromise. The speaker also names a second construction path: find a bad model response in logs, then rewrite it into the desired answer. This produces the same kind of pair but requires more human or expert work because someone must author the preferred response. The data decision therefore balances cost, label quality, and behavioral specificity. The preferred practical unit is not an isolated score; it is a same-prompt comparison that says which completion should win.
CoreFocus: Pairwise comparison reduces calibration burden because the rater judges two concrete responses under one prompt.
Concept: LLM-as-judge - a model-based evaluator that compares candidate outputs, often used when human rating is too expensive or slow.
Application: Pairwise labeling also helps because preferences often depend on context rather than universal score scales. A response that is good for a safety question may be too cautious for creative writing, and a friendly answer may be too verbose for a code task. Same-prompt comparison keeps the judgment local to the actual decision the model must make.
Recipe slide for getting pairwise preference data
Notes: The lecture treats preference labels as behavioral supervision. They can measure helpfulness, friendliness, safety, or another chosen dimension, but the dimension must be defined.
LinkBack: 15:00-20:00, binary preference labels, human and model judges, rewrite-based data construction.
005 · RL formulation for LLMs: agent, state, action, policy, reward
20:00-26:00
The lecture maps LLM generation into reinforcement-learning language. The agent is the LLM. The state is the input context the model has so far. The action is the next token the model chooses to generate. The action space is the vocabulary. The policy is the next-token probability distribution produced by the forward pass. This mapping can look strange because an LLM is not moving through a physical environment, but it is still making sequential decisions: every generated token changes the context for the next decision. Preference tuning adds the missing RL ingredient, the reward. A generated completion receives some signal that says how good it was, and that signal feeds back into the policy parameters. The speaker also answers a student question about whether the reward is per token or for the whole output. The high-level setup treats the output quality as a completion-level signal, although PPO-style algorithms later distribute credit through advantage estimates and token probabilities. In the RL view, LLM tuning changes the policy that maps a context to next-token probabilities.
CoreFocus: State is context, action is next token, and policy is the model's probability distribution over vocabulary.
Concept: Policy - the conditional probability distribution over next tokens, parameterized by the LLM weights.
Mechanism: Once generation is described as a policy, every alignment method can be read as a way of changing probability mass. The model does not store a single answer; it stores a distribution over next tokens that compounds into a distribution over full responses. Preference tuning works by reshaping that distribution toward preferred trajectories.
RL formulation slide for LLMs
Notes: This mapping matters because later formulas operate on probability ratios between policies, not directly on natural-language strings.
LinkBack: 20:00-26:00, RL formulation for LLMs and policy alignment objective.
006 · RLHF as two stages: reward model then policy optimization
26:00-32:00
With the RL mapping in place, the lecture defines RLHF: reinforcement learning from human feedback. The speaker stresses that the name is historically broad: the feedback may come from humans, LLM judges, or other evaluators, but the canonical phrase remains RLHF. The pipeline has two stages. Stage one learns how to distinguish good outputs from bad outputs. It takes the collected preference pairs and trains a reward model. The input to this model is the concatenation of the prompt and response; the output is a scalar score. Stage two uses that reward model to tune the LLM policy. Given a prompt, the current LLM generates a completion, the reward model scores that prompt-completion pair, and the training algorithm updates the LLM so future completions should score better. The lecture then introduces the Bradley-Terry formulation as the mathematical bridge from pairwise preferences to reward scores. If response i should beat response j, the probability of that event depends on the difference between their reward scores. RLHF works by first learning a preference scorer, then using that scorer as the reward signal for policy optimization.
CoreFocus: The reward model is the hinge: it converts qualitative comparisons into the scalar signal required by RL.
Concept: RLHF - a two-stage alignment recipe: reward modeling from preferences followed by reinforcement learning on the language-model policy.
Mechanism: RLHF separates evaluation from generation. The reward model learns to evaluate prompt-response pairs, while the policy model learns to generate responses that receive high evaluations. This separation is powerful but brittle because any error in the evaluator becomes training signal for the generator. The policy is only as aligned as the reward model is faithful to the intended preference.
Bridge: **This two-stage design is also why RLHF is easy to describe but hard to debug. A policy failure may come from bad preference data, a weak reward model, unstable RL updates, or a mismatch between the reward dimension and the user need.
**
Step 1 reward modeling slide
Notes: The reward model does not need to generate text; it evaluates a prompt-response pair and produces a scalar proxy for a preference dimension.
LinkBack: 26:00-32:00, RLHF definition, two-stage pipeline, reward model setup.
007 · Bradley-Terry reward modeling and the pairwise loss
32:00-40:00
The Bradley-Terry model expresses the probability that one response beats another as a soft comparison between their scores. In lecture notation, the probability that output yi is preferred to output yj is exp(score_i) divided by exp(score_i) plus exp(score_j), equivalently sigmoid(R_i minus R_j). The sigmoid matters because only the difference in reward scores should determine the pairwise probability. If the winning response has a much higher score than the losing response, the sigmoid approaches one. If the scores are reversed, it approaches zero. Training the reward model therefore asks for parameters that make observed winning responses score higher than observed losing responses. The speaker reconstructs the loss from maximum likelihood. Given independent preference pairs, maximize the product of the probabilities assigned to all observed winner-loser events. Products of probabilities can become numerically small, so the objective becomes a sum of log probabilities. ML practice then changes maximization into minimization by adding a negative sign, yielding a negative log-likelihood over pairwise comparisons. The reward model is trained so that the reward difference R(prompt, winner) - R(prompt, loser) makes the observed preference likely.
CoreFocus: Bradley-Terry turns pairwise preference into a differentiable probability through the sigmoid of a reward difference.
Concept: Pairwise NLL - the negative log-likelihood loss that penalizes the reward model when the preferred response does not score above the rejected response.
Mechanism: The Bradley-Terry model is useful because it does not require a calibrated absolute reward target. It only requires the score gap to explain the observed ordering inside each pair. If the chosen answer wins by a large margin, the loss pushes the reward gap larger; if the gap is wrong, the loss strongly corrects it. The supervision is comparative, so the learned reward scale is meaningful mainly through differences.
Bridge: **For studying the formula, keep the old-new flow clear: start with a labeled pair, turn each response into a reward score, subtract loser score from winner score, pass that gap through a sigmoid, then optimize the model so observed winners become probable.
**
Reward-model loss formula derived from preference pairs
Notes: Absolute reward scale is less important than relative ordering inside pairs; this is why adding a constant to all scores does not change a pairwise comparison.
LinkBack: 32:00-40:00, Bradley-Terry, sigmoid, maximum likelihood, negative log-likelihood.
008 · Reward model architectures and preference dimensions
40:00-47:00
After the loss, the lecture asks what model should produce the reward score. Because modern LLM systems already use decoder-only models, one common route is to take a decoder-only LLM and attach a reward or classification head at the end of the sequence. An encoder-only model such as BERT could also be used by projecting the CLS-token embedding, but the lecture notes that the decoder-only route is common because everything in this ecosystem is now an LM. The output is a continuous score: a good response might receive a positive value and a poor response a negative value. The speaker also answers a question about whether the reward is universal. It is not. Reward models are usually trained with respect to a dimension such as usefulness, friendliness, harmlessness, safety, or a holistic blend. That design choice matters because an answer can be useful but unfriendly, safe but unhelpful, or factually correct but too harsh. RewardBench is mentioned as a benchmark for evaluating reward models. A reward model only measures the preference dimension it was trained to measure, so reward design defines what alignment means.
CoreFocus: The reward score is continuous, but its meaning comes from the labeled comparison dimension.
Concept: Reward dimension - the behavioral axis being scored, such as helpfulness, safety, friendliness, or overall user preference.
Application: Reward dimensions should be treated like product specifications. A harmlessness reward, a helpfulness reward, and a friendliness reward may all score the same answer differently, and a deployed assistant may need a blend or hierarchy among them. Before optimizing a reward model, the builder must decide what kind of goodness the reward is supposed to measure.
Bridge: **The reward-model architecture is less important than the contract it serves. Whatever network is used, it must preserve the prompt-response relationship and output a scalar whose ordering agrees with the preference labels for the chosen behavioral dimension.
**
Reward modeling slide with model choices and reward dimensions
Notes: Do not read the scalar reward as ground truth utility; read it as a learned proxy for one defined preference criterion.
LinkBack: 40:00-47:00, reward-model architecture, scalar outputs, reward dimensions, RewardBench.
009 · Using the reward model: rollout, score, update, and reward hacking
47:00-54:00
Stage two uses the reward model to train the policy. The LLM receives a prompt and generates a completion, also called a rollout. The prompt and rollout go into the reward model, which returns a score. That score then guides the update of the LLM weights. This loop looks direct, but the lecture immediately asks what can go wrong if the model maximizes the reward too aggressively. Students identify overfitting to noisy data, and the speaker names the central failure mode: reward hacking. Reward hacking occurs when the reward is an imperfect proxy for the true objective. The lecture's analogy is a teacher who wants an informative lecture but chooses applause volume as the reward. If jokes produce more applause than careful explanation, maximizing applause can make the lecture less informative even while the measured reward improves. In LLM tuning, the reward model can similarly push the policy toward patterns that satisfy the proxy but miss the real human goal. Reward hacking is proxy optimization: the score improves because the model exploits the reward, not because the behavior truly improves.
CoreFocus: A reward model is useful only as a proxy; treating it as the true objective creates alignment risk.
Concept: Rollout - a full model-generated completion sampled from the current policy during RL training.
Mechanism: Reward hacking appears because optimization discovers shortcuts faster than humans discover all proxy failures. Once the policy learns how the reward model reacts, it can favor patterns that the reward model overvalues. That is why the lecture immediately pairs reward maximization with constraints and evaluation. A learned reward should guide behavior, not become an unquestioned definition of success.
Bridge: **The reward-hacking warning should be read as a systems warning, not just a moral warning. Any proxy that enters an optimizer becomes an object the model can exploit, so evaluation and regularization must surround the reward objective.
**
Step 2 reinforcement learning slide using reward model scores
Notes: The optimization target must include constraints because unconstrained reward maximization can move the model into unnatural or exploitative completion patterns.
LinkBack: 47:00-54:00, RL step, rollout scoring, reward hacking example.
010 · KL divergence: keep the tuned policy near the base model
54:00-61:00
To prevent policy drift, the lecture introduces a second term in the RL objective: the tuned policy should not move too far from a base or reference model. The mathematical measure used is KL divergence. The speaker is careful not to call it a distance because it lacks some distance properties, but it measures how far one probability distribution is from another. For discrete distributions P and Q, KL uses a sum of p_i log(p_i/q_i). It is nonnegative and equals zero only when the distributions are equal, a result connected to Jensen's inequality. In the LLM setting, the distributions are next-token distributions or policy distributions. The objective therefore balances two forces: increase the reward score, but penalize large divergence from the base SFT model. This explains why preference tuning is not simply make the reward as high as possible. If the policy moves too far, it may lose fluency, safety, factual habits, or broad behavior learned during pretraining and SFT. The KL term acts as an anchor: optimize preference, but do not let the policy forget the model it came from.
CoreFocus: KL regularization turns preference optimization into a controlled update rather than an unconstrained search for reward.
Concept: Reference policy - the fixed base model, usually the SFT model, used to measure how far the tuned policy has moved.
Mechanism: KL regularization preserves useful inherited behavior by comparing probability distributions, not final text strings. Two completions may look similar while their token distributions have moved substantially, and small token-probability changes can compound across long generations. The reference model anchors the whole policy distribution, not merely a few visible sample outputs.
Bridge: **KL also explains why alignment training is conservative by design. The goal is not to discover an arbitrary high-reward language model from scratch, but to make a bounded movement from a useful SFT model toward preferred completions.
**
KL divergence and reward-KL objective slide
Notes: The beta coefficient in reward-minus-KL objectives controls the trade-off between reward improvement and staying close to the reference model.
LinkBack: 54:00-61:00, KL divergence definition, nonnegativity, reward-KL trade-off.
011 · Value function, advantage, and PPO-Clip
61:00-69:00
PPO introduces more structure because token-level policy updates need a way to decide which sampled actions should be reinforced. The lecture defines a value function as an estimate of the final reward if the model continues generating according to its policy. In implementation, an LLM initialized from the SFT model can have an additional value head trained jointly with the policy. The advantage then compares how good a sampled action or trajectory was relative to what the value function expected. A positive advantage says the behavior should be reinforced; a negative advantage says it should be suppressed. PPO-Clip uses a ratio r(theta) between the probability under the current policy and the probability under the previous policy. If the advantage is positive, increasing this ratio increases the objective, but only up to a clipping threshold. If the advantage is negative, the objective discourages the action, again without allowing extreme updates. The speaker emphasizes that the old policy in this ratio is the previous RL iteration, not the original SFT base model. PPO-Clip uses the advantage to choose update direction and clipping to prevent one RL step from changing the policy too much.
CoreFocus: Advantage tells whether to reinforce a sampled output; the probability ratio tells how large the policy update is.
Concept: Advantage - a signal, often built from reward and value estimates, that says whether an action performed better or worse than expected.
Mechanism: PPO needs the advantage because a final scalar reward does not directly say which token decisions caused success. The value function gives a baseline expectation, and the advantage converts reward into a relative signal for sampled behavior. Clipping then prevents that signal from making a destabilizing jump. PPO is a controlled credit-assignment method for policy updates.
Bridge: **PPO therefore combines three ideas: reward says what looked good, value and advantage estimate how surprisingly good it was, and clipping limits how much the policy can change from that evidence in one update.
**
PPO-Clip objective slide with clipped policy ratio
Notes: PPO is maximizing an objective even though ML libraries often implement it as minimizing a negative loss; this sign convention is easy to confuse.
LinkBack: 61:00-69:00, value head, generalized advantage intuition, PPO-Clip ratio and clipping.
012 · PPO-KL, model copies, and RL variants
69:00-76:00
The lecture then presents a PPO variant with an explicit KL penalty. In PPO-KL, the objective includes the ratio times advantage, minus a KL divergence between policies. The speaker compares this with the clipped version: both are trying to prevent large policy updates, but one enforces it through clipping and the other through a penalty. The LLM version also needs to compare the current policy with the frozen SFT reference model, not only with the previous iteration, because the final tuned model should remain close to the base assistant. This explains why RLHF can require many model components at once: the current policy being optimized, the old policy from the previous iteration, the reward model, the value function or value head, and the frozen reference/base model. The lecture then widens the view: PPO has been popular, but there are variants and other algorithms, including GRPO, which will matter for reasoning models in the next lecture. The operational burden of RLHF comes from stabilizing policy improvement while carrying multiple policy, reward, value, and reference components.
CoreFocus: PPO is not one magic formula; it is a stability framework around reward-driven policy updates.
Concept: PPO-KL - a PPO variant that directly penalizes KL divergence instead of only relying on clipped probability ratios.
Application: The model-copy burden explains why RLHF feels different from ordinary fine-tuning in engineering practice. The trainer must coordinate the trainable policy, the previous policy, the frozen reference, the reward evaluator, and the value estimator while tracking KL and reward behavior. The complexity is systemic, not just mathematical.
PPO-KL objective and model-copy discussion slide
Notes: The lecture's model-count discussion explains why RLHF is not just another small fine-tuning job; it is a training system with several coupled estimators.
LinkBack: 69:00-76:00, PPO-KL, reference model, model-copy burden, GRPO preview.
013 · RLHF challenges and on-policy training
76:00-83:00
The speaker summarizes why RL-based tuning is hard in practice. First, it is a two-stage process: train the reward model, then train the policy. If the reward model has a problem, the policy-training stage may have to be redone. Second, many hyperparameters matter, including KL penalties, learning rates, clipping thresholds, rollout settings, and reward-model behavior. Third, the process is computationally heavy because it generates outputs during training and evaluates them. The lecture then defines on-policy training. In SFT, training examples are fixed prompt-response pairs; the current model is not required to generate the answer during the training example. In PPO-style RLHF, the current policy samples completions at each iteration, and the update uses those current-policy samples. That is on-policy. The contrast matters because on-policy training can adapt to the model's current behavior, but it also raises cost and instability. A student asks why not simply do SFT on preference data. The answer is that SFT can imitate a target response, but it cannot naturally say do not generate this rejected response unless the data is rewritten into demonstrations. On-policy RLHF is expensive because the model must generate the behavior it is about to be judged on.
CoreFocus: SFT trains from fixed demonstrations; PPO trains from current-policy rollouts and preference-derived rewards.
Concept: On-policy - training on data sampled from the model being updated, rather than from a fixed external dataset.
Mechanism: On-policy training is attractive because it judges the model where it currently lives. As the policy changes, its mistakes and opportunities change, so new rollouts expose new behavior to optimize. The cost is that data collection is now inside the training loop. The model must generate, be judged, and update repeatedly rather than learn from a static table.
Slide listing challenges with RL-based approaches
Notes: On-policy training helps target the model's actual behavior, but it makes each training iteration more like a miniature deployment-and-evaluation loop.
LinkBack: 76:00-83:00, two-stage dependency, hyperparameters, on-policy versus off-policy, SFT limitation.
014 · Best-of-N: avoiding training by paying at inference
83:00-91:00
Before introducing DPO, the lecture discusses a simpler alternative: best-of-N. The method samples several completions for the same prompt, scores each with a reward model or evaluator, and returns the highest-rated candidate. This avoids an RL training loop, so it can look attractive. But the speaker pushes the class to identify the cost. If the base model is poor, many samples may still be poor. More importantly, inference becomes expensive because the system must query the model N times for every user request and then score the candidates. Even with enough compute to run the samples in parallel, latency can worsen because the user may wait for the slowest sampled completion. The distribution of the maximum completion time shifts right as N grows. The method therefore moves cost from training to serving. Whether this is acceptable depends on traffic, latency requirements, compute budget, and how often the system is used. Best-of-N is not free alignment; it trades RL training complexity for repeated inference cost and worse tail latency.
CoreFocus: The same reward model can rank outputs at inference, but serving economics may dominate the technical convenience.
Concept: Reranking - generating multiple candidates and choosing the best according to a scoring function after generation.
Application: Best-of-N is best understood as runtime selection rather than model improvement. It can improve one returned answer without changing the model parameters, which is useful for low-volume or high-value prompts. But for a high-traffic assistant, multiplying generation count multiplies serving cost. The method buys quality per request by spending more inference on that request.
Best-of-N action slide with reward scores
Notes: Use best-of-N when inference volume is low or quality is worth repeated sampling; avoid it when high-traffic serving cost and latency are central.
LinkBack: 83:00-91:00, best-of-N example, inference cost, latency tail, training-versus-serving trade-off.
015 · Why DPO is introduced: complaints about RLHF
91:00-97:00
The lecture introduces DPO by first collecting the complaints against PPO-style RLHF. In PPO, the loss and update machinery implicitly carry several model weights or estimators: the current policy, the old policy or reference policy, the reward model, and the value function through the advantage. Best-of-N avoids that training burden but creates inference burden. DPO is motivated by the desire to keep preference learning but remove the explicit RL machinery. The key question becomes whether one can use preference pairs directly, without first training a reward model and then using that reward model inside PPO. The answer comes from the reward-plus-KL objective already discussed. If the optimal policy under that objective can be expressed analytically, then the reward can be rewritten in terms of the policy and a reference model. That makes it possible to substitute this expression into the same Bradley-Terry preference probability used for reward modeling. DPO starts from the RLHF objective but asks whether pairwise preference data can supervise the policy directly.
CoreFocus: The DPO motivation is practical: preserve the preference signal while removing reward-model and PPO training complexity.
Concept: Direct preference supervision - fitting a model to winner-loser preference pairs without training a separate scalar reward model first.
Mechanism: DPO becomes appealing because it attacks the same preference problem at the loss level. Instead of learning a reward model and then running an RL optimizer, it asks whether the preference pair can directly push the policy in the desired direction relative to a reference. The simplification is to make the pairwise comparison itself the training objective.
Bridge: **This is the conceptual entry point to DPO: if the hard part of RLHF is the machinery around reward modeling and policy optimization, then a direct loss is attractive exactly because it preserves pairwise supervision while deleting moving parts.
**
Slide comparing supervised approach with DPO
Notes: DPO is not a new kind of preference data; it is a different way to turn the same pairwise data into a training objective.
LinkBack: 91:00-97:00, PPO model burden, best-of-N limitation, setup for DPO.
016 · DPO derivation: rewrite reward through policy ratios
97:00-103:00
The mathematical move behind DPO begins with the same objective used to motivate KL-constrained RL: maximize reward while penalizing divergence from the reference policy. The DPO paper solves for the optimal policy pi-star under that objective. The resulting expression contains the reward, the reference policy, a beta coefficient that controls divergence strength, and a partition function that normalizes the distribution. The lecture emphasizes that this step adds no new modeling assumption; it is algebra on the existing objective. Rearranging the expression lets the reward be written as a function of the optimal policy and the reference policy. Then the key insight is to plug this reward expression into the Bradley-Terry model for pairwise preferences. Because Bradley-Terry only needs reward differences between a winning and losing response, terms that are shared or normalizing can cancel or become irrelevant to the loss. This produces an objective that compares how much more likely the trainable policy makes the winning response than the losing response relative to the reference model. DPO removes the explicit reward model by replacing reward differences with policy-reference log-probability differences.
CoreFocus: The reward is not ignored; it is implicitly represented through how the optimized policy should differ from the reference policy.
Concept: Policy-reference ratio - the comparison between the trainable policy's probability and the fixed reference model's probability for a response.
Mechanism: The derivation matters because it keeps DPO connected to the reward-plus-KL objective rather than presenting it as an unrelated heuristic. The reference model still controls drift, beta still controls the strength of that control, and the winner-loser comparison still comes from Bradley-Terry. DPO is a supervised-looking loss with an RLHF objective hidden inside its algebra.
Bridge: **When reading the DPO derivation, do not memorize algebra first. Track the role of each term: the trainable policy should favor winners, the reference policy controls drift, and beta scales how aggressively preference pressure can move the model.
**
DPO derivation slide from reward-KL objective
Notes: The beta coefficient still matters because it controls how strongly the learned policy may depart from the reference model while satisfying preferences.
LinkBack: 97:00-103:00, optimal policy expression, reward rearrangement, Bradley-Terry substitution.
017 · DPO objective and the two-model simplification
103:00-107:00
After substitution, DPO directly fits a loss over preference pairs. For each prompt, the training example contains a chosen response and a rejected response. The model is trained so the chosen response becomes more likely than the rejected response in the right policy-reference sense. Operationally, this is much simpler than PPO-style RLHF. Instead of carrying a current policy, an old policy, a reward model, and a value function, DPO mainly needs the trainable policy and a fixed reference policy, often the base SFT model. The lecture uses this to answer the practical question: if DPO is easier, why does everyone not always use it? The answer is that simplicity has trade-offs. DPO can fit preference data directly, but if that data distribution differs from the model's own generation distribution, the optimization may not match the same behavior PPO would see during on-policy rollouts. Some studies find PPO can outperform DPO after careful tuning, while DPO can get close with much less engineering effort. DPO wins on simplicity because it turns preference tuning into a supervised pairwise loss with only policy and reference models.
CoreFocus: DPO replaces the reward-model-plus-RL stack with direct preference optimization against a frozen SFT reference.
Concept: Distribution shift - mismatch between the preference data used for direct training and the completions the model would generate during its own policy rollouts.
Application: The practical PPO-versus-DPO decision is therefore not ideological. DPO is attractive when iteration speed, simplicity, and reduced infrastructure matter. PPO is attractive when the team can tune RL carefully and the application justifies extra training complexity for final performance. The right method depends on whether engineering simplicity or last-mile optimization is the binding constraint.
Slide comparing PPO-based RLHF and DPO behavior
Notes: DPO is often attractive when the goal is a strong, quick preference-tuning run; PPO may be worth the cost when expert RL tuning and last-mile performance matter.
LinkBack: 103:00-107:00, DPO loss, model-count reduction, DPO versus PPO trade-offs.
018 · Behavioral meaning: preference tuning changes the chosen completion
107:00-107:41
The lecture closes by returning to behavior. In the teddy-bear washing example, the rejected answer may be factually correct: teddy bears often should be handwashed rather than put in a harsh washing-machine cycle. But factual correctness is not the whole preference criterion. The answer can still be too rough, too unfriendly, or poorly matched to a user who cares about the teddy bear. A preferred answer can communicate the same factual caution while sounding gentle: it is better not to wash the teddy bear that way because it could get hurt; gentle handwashing is safer. This final example makes the lecture's alignment thesis precise. Preference tuning is not mainly about adding new facts to the model. It works on the distribution of plausible completions the model already can produce and shifts probability mass toward the completion humans prefer under the chosen criterion. The final practical rule is method selection. Use DPO when quick, strong preference tuning with fewer moving parts is sufficient. Use PPO-style RL when you have RL expertise, can afford the training complexity, and need the last bit of performance. Preference tuning changes which plausible answer the model returns, and that choice is where alignment lives.
CoreFocus: The preferred answer can preserve the fact while changing tone, helpfulness, and user fit.
Concept: Behavioral alignment - selecting completions that satisfy human preference criteria such as helpfulness, gentleness, safety, and appropriateness.
Mechanism: The final teddy-bear contrast also explains why alignment data is subtle. A factual correction can be delivered as a rejection, a warning, a suggestion, or a caring explanation. Preference tuning chooses among those communication modes. The model is not merely choosing facts; it is choosing how those facts meet the user.
Behavior slide showing factually correct but unfriendly teddy-bear answer
Notes: The closing comparison is the cleanest intuition for the whole lecture: tuning often changes style, ranking, and preference satisfaction rather than adding knowledge.
LinkBack: 107:00-107:41, teddy-bear behavior example, PPO versus DPO practical choice, closing takeaway.
5. Suggested Answers
Question 1. Why does Lecture 5 introduce preference tuning after SFT instead of treating SFT as enough alignment?
Answer 1. SFT teaches the model to imitate desired prompt-response examples, but it does not naturally encode rejected outputs or subtle comparisons between plausible completions. Preference tuning adds relative behavioral supervision: for the same prompt, one answer should win and another should lose.
Question 2. What makes pairwise preference labels easier and more useful than absolute pointwise scores for LLM outputs?
Answer 2. Pointwise scoring asks a rater to calibrate an absolute number across prompts and tasks, which is subjective and unstable. Pairwise labeling only asks which of two same-prompt responses is better, so the judgment is local, concrete, and easier to collect.
Question 3. In the RL formulation for LLMs, what are the agent, state, action, policy, and reward?
Answer 3. The agent is the LLM, the state is the context so far, the action is the next token, the policy is the next-token probability distribution, and the reward is a preference-derived score for the generated output. The RL formulation treats generation as sequential policy decisions over tokens.
Question 4. How does RLHF turn pairwise human preferences into a scalar reward model?
Answer 4. RLHF first trains a model that takes a prompt-response pair and outputs a scalar score. Winner-loser preference pairs train that score so preferred responses receive higher rewards than rejected responses. The reward model converts qualitative comparisons into the numeric signal that RL can optimize.
Question 5. Why does the Bradley-Terry formula use a sigmoid of the reward difference rather than two independent scores?
Answer 5. Pairwise preference depends on which response scores higher relative to the other, not on either score in isolation. The sigmoid of R_winner minus R_loser maps a score difference into the probability that the winner should beat the loser.
Question 6. What loss does the reward model minimize, and how does it come from maximum likelihood on winner-loser pairs?
Answer 6. Assuming preference pairs are independent, the reward model maximizes the product of the Bradley-Terry probabilities assigned to observed winner-loser events. Taking logs turns the product into a sum, and ML convention minimizes the negative of that sum. The loss is negative log-likelihood over pairwise preferences.
Question 7. Why can reward hacking occur even when the optimizer is successfully increasing the reward-model score?
Answer 7. The reward model is only a proxy for the real objective. If the proxy is imperfect, the policy can learn behaviors that raise the score while missing the intended goal, like maximizing applause instead of lecture informativeness. Reward hacking is success on the metric and failure on the objective.
Question 8. Why does RLHF need a KL term or reference model instead of simply maximizing reward as hard as possible?
Answer 8. Unconstrained reward maximization can push the model away from the useful behavior learned during pretraining and SFT. KL regularization compares the tuned policy with a reference model and penalizes excessive drift. The KL term anchors preference improvement to the base model's distribution.
Question 9. What role do the value function and advantage play in PPO-style tuning?
Answer 9. The value function estimates expected future reward under the policy, and the advantage measures whether a sampled behavior was better or worse than expected. Advantage tells PPO which sampled outputs to reinforce and which to suppress.
Question 10. Why does PPO clip or penalize the ratio between the current and old policy?
Answer 10. The ratio measures how much the current policy changes the probability of sampled actions relative to the previous policy. PPO clips or penalizes that ratio to prevent a single update from moving too far and destabilizing training. PPO limits update size while still using reward information to improve the policy.
Question 11. What makes PPO-style RLHF operationally expensive compared with SFT?
Answer 11. PPO-style RLHF needs rollouts from the current model, reward-model scoring, value or advantage estimation, reference-policy comparison, and careful hyperparameter tuning. It is a multi-model training system rather than a fixed-dataset imitation run.
Question 12. Why is PPO called on-policy training, and why is SFT not on-policy in the same sense?
Answer 12. PPO samples completions from the current policy and updates the model based on those samples. SFT trains on fixed demonstration responses that usually were not generated by the current model during that iteration. On-policy training judges the model on behavior it just produced.
Question 13. Why does best-of-N avoid RL training but still create deployment cost and latency problems?
Answer 13. Best-of-N samples several completions and returns the highest-scored one, so it does not need to update the policy. But every user request may require N generations plus scoring, and the response may wait for the slowest candidate. Best-of-N moves alignment cost from training into inference.
Question 14. What is the key mathematical move that lets DPO remove the explicit reward model?
Answer 14. DPO starts from the reward-plus-KL objective, solves for the optimal policy, rearranges the expression so reward is written through policy and reference probabilities, and inserts that reward expression into Bradley-Terry. Reward differences become policy-reference log-probability differences.
Question 15. Why might PPO outperform DPO in some settings even though DPO is simpler?
Answer 15. PPO trains on current-policy rollouts and can more directly optimize reward under the model's generated distribution, though it is harder to tune. DPO fits preference pairs directly and can suffer when those pairs do not match the model's generation distribution. PPO may buy extra performance with on-policy complexity, while DPO buys simplicity with a distribution-shift trade-off.
Question 16. In the teddy-bear example, why is the preferred answer not mainly about factual correctness?
Answer 16. The rough answer may be factually correct if handwashing is safer, but the preferred answer communicates the same caution with gentleness and user empathy. Preference tuning often changes tone and response selection, not the underlying fact.