LLM training

1. Summary

Meta: Source type: YouTube lecture; course: Stanford CME295 Large Language Models; lecture: Lecture 4 - LLM Training; speaker/channel: Stanford Online; language: English; duration: about 1:47:23; transcript type: timestamped SRT; generated date: 2026-09-10.

Core: The lecture explains LLM training as a staged systems problem: pretraining learns next-token structure at enormous compute scale, engineering methods make that training feasible, and fine-tuning plus alignment reshape the pretrained model into a useful assistant.

Abstract: Lecture 4 first closes logistics for the upcoming midterm and reviews the prior lecture's MoE, decoding, prompting, and inference-efficiency material. The main technical arc then starts with pretraining: a decoder-only model learns from internet-scale corpora by predicting the next token, and its cost scales roughly with tokens, parameters, and hardware speed. The lecture uses scaling laws, GPU memory limits, data/model parallelism, FlashAttention, and mixed precision to show why training is constrained by both arithmetic and memory movement. The second half shifts from learning language to becoming helpful: supervised fine-tuning trains on input-output pairs, instruction tuning targets assistant behavior, evaluation exposes benchmark and preference limitations, and alignment combines fine-tuning with later preference tuning. The final section explains LoRA and QLoRA as parameter-efficient fine-tuning methods that freeze the pretrained model and learn small task-specific updates. The main study move is to treat LLM training as a lifecycle: learn broad distribution, distribute the cost, adapt behavior, evaluate usefulness, then tune cheaply.

Concept: Pretraining - the initial expensive stage where a decoder-only LLM learns next-token prediction from very large, broad corpora.

Concept: Training FLOPs - the approximate number of floating-point operations needed for training, distinct from FLOP/s as hardware throughput.

Concept: Chinchilla scaling - the empirical compute-optimal trade-off between model parameters and training tokens for a fixed compute budget.

Concept: Data parallelism - distributing different batch shards across devices while each device holds a model copy or a sharded model state.

Concept: Model parallelism - splitting model computation or parameters across devices through tensor, pipeline, sequence, context, or expert parallelism.

Concept: FlashAttention - an exact attention implementation that reduces high-bandwidth-memory traffic by tiling attention computation through faster on-chip memory.

Concept: Mixed precision training - using lower precision for most forward/backward computation while keeping critical model weights or updates in higher precision.

Concept: Supervised fine-tuning - further training a pretrained model on labeled input-output pairs so it responds usefully to prompts.

Concept: Instruction tuning - an SFT subcase where data teaches the model to answer user instructions as an assistant.

Concept: Alignment - the post-pretraining process, here framed as fine-tuning plus preference tuning, that pushes model behavior toward user and product goals.

Concept: LoRA - a parameter-efficient fine-tuning method that freezes pretrained weights and learns a low-rank additive update.

Concept: QLoRA - LoRA with quantized frozen base weights, often using NF4 and double quantization to reduce VRAM while training the low-rank adapters.

TakeHome: Pretraining creates broad language competence, but it does not by itself create an assistant that understands what the user wants.

TakeHome: Training feasibility depends as much on memory placement, precision, and parallelism as on the transformer architecture.

TakeHome: SFT and LoRA exploit the pretrained model's broad representation: they redirect behavior without paying the full cost of training a new LLM from scratch.

2. Knowledge Tree

CME295 Lecture 4 - LLM Training/
├── Course logistics and recap/
│   ├── midterm scope: lectures 1-4
│   ├── prior lecture: MoE, decoding, prompting, KV cache
│   └── lifecycle preview: pretraining, fine-tuning, preference tuning
├── Pretraining objective and scale/
│   ├── broad corpus: web, code, multilingual text
│   ├── decoder-only next-token prediction
│   ├── loss on predicted tokens
│   ├── training FLOPs versus hardware FLOP/s
│   └── Chinchilla-style token-parameter balance
├── Distributed training systems/
│   ├── GPU and TPU matrix multiplication
│   ├── weights, activations, gradients, optimizer states
│   ├── GPU memory limit
│   ├── data parallelism and ZeRO variants
│   └── model parallelism: expert, tensor, pipeline, sequence, context
├── Efficient training kernels and arithmetic/
│   ├── GPU memory hierarchy: HBM versus SRAM
│   ├── self-attention matrix structure
│   ├── FlashAttention tiling
│   ├── recomputation trade-off
│   └── mixed precision: FP32, FP16, BF16, quantization intuition
├── Fine-tuning and instruction behavior/
│   ├── pretrained model as language model, not assistant
│   ├── SFT input-output pairs
│   ├── loss only over desired output tokens
│   ├── instruction data mixture
│   ├── helpfulness, harmlessness, safety refusal, hedging
│   └── distribution coverage and generalization
├── Evaluation and alignment/
│   ├── benchmark scores and contamination risk
│   ├── task accuracy versus user value
│   ├── Chatbot Arena pairwise preference
│   ├── voter, style, safety, and population biases
│   ├── preference tuning
│   └── mid-training between pretraining and SFT
└── Parameter-efficient adaptation/
    ├── LoRA: W = W0 + BA
    ├── frozen base weights
    ├── small rank r adapter matrices
    ├── attention and feed-forward insertion choices
    ├── rank as a design hyperparameter
    └── QLoRA: NF4 quantization and double quantization

3. Feynman Questions

Question 1. Why does the lecture separate pretraining from fine-tuning instead of treating all LLM training as one next-token prediction run?

Question 2. What does the pretraining objective actually optimize in a decoder-only LLM, and where does the loss get applied?

Question 3. Why are FLOPs and FLOP/s different quantities, and why does the lecture need both when discussing LLM training cost?

Question 4. What does the Chinchilla-style scaling argument say about the trade-off between parameter count and token count?

Question 5. Why can a model like GPT-3 be described as undertrained under compute-optimal scaling logic?

Question 6. Why does training require storing more than the model weights, and why does that immediately create a GPU-memory bottleneck?

Question 7. How does data parallelism differ from model parallelism in the way it splits work across GPUs?

Question 8. What problem do ZeRO-style methods solve inside data parallel training?

Question 9. Why does FlashAttention speed up exact attention without changing the mathematical attention result?

Question 10. Why can FlashAttention recompute activations and still run faster than the vanilla implementation?

Question 11. Why does mixed precision keep some quantities in high precision while moving other operations to lower precision?

Question 12. Why is a pretrained next-token model not automatically a helpful assistant in the teddy-bear washer example?

Question 13. In SFT, why does the loss start on the answer side rather than on the user instruction itself?

Question 14. What kinds of data enter instruction tuning, and why does safety data belong in the same behavioral training story?

Question 15. Why can benchmark scores and Chatbot Arena rankings fail to capture the value of a model for a specific user or domain?

Question 16. How does the lecture define alignment, and where does mid-training fit relative to pretraining and fine-tuning?

Question 17. What is the low-rank update in LoRA, and why does freezing W0 save fine-tuning cost?

Question 18. Why does QLoRA quantize the frozen base weights but keep the trainable LoRA matrices in higher precision?

4. Narrative Study Notes

001 · Logistics and lecture scope

00:00-05:00

The lecture opens with course logistics because the same material also defines what the learner must be able to reconstruct for the midterm. The exam is scheduled for the following week, runs for 90 minutes, and covers lectures 1 through 4. The speaker frames the exam as mostly grounded in class material: recordings, slides, and important formulas. That logistical point matters technically because Lecture 4 is not a loose add-on; it is the training chapter that completes the first exam arc after transformers, transformer variants, and LLM behavior. The lecture then previews the transition from prior architecture and inference topics into how a model is trained, optimized, and adapted. For studying, Lecture 4 is the bridge from what an LLM is to how its weights, data, hardware, and post-training stages make that model usable.

CoreFocus: Treat the opening logistics as a scope statement: lectures 1-4 form the exam unit, and this lecture supplies the training systems layer.

Concept: Training lifecycle - the ordered path from initialized model to pretrained model, fine-tuned model, preference-tuned model, and practical assistant. Lecture logistics slide with midterm scope

Notes: Write down the course-level dependency: architecture and inference methods from prior lectures become the objects that Lecture 4 will train and adapt.

LinkBack: 00:00-05:00, midterm logistics and lecture roadmap.

002 · Recap: architecture, decoding, and inference before training

05:00-10:00

Before introducing training, the speaker reconstructs the prior lecture's conceptual map. MoE models use a gate or router to activate only selected experts, which separates total parameter capacity from active computation. Decoder-only LLMs output a next-token distribution, and decoding turns that distribution into actual text through greedy selection, beam search, sampling, and temperature scaling. The recap also names inference optimizations such as KV caching, grouped-query attention, PagedAttention, speculative decoding, and multi-token prediction. This review is not repetition; it places training in the same systems frame as inference. A model must first learn a next-token distribution, and then deployment-time machinery decides how efficiently and usefully that distribution is used. Training explains where the distribution comes from; decoding and inference explain how that learned distribution becomes a response under resource constraints.

CoreFocus: The lecture reuses the same decoder loop from Lecture 3, but now asks how the loop's probabilities are learned in the first place.

Concept: Decoder loop - the autoregressive process in which the model maps a prefix to logits, converts logits to probabilities, and chooses the next token. Lecture recap slide connecting transformer models and training optimizations

Notes: Keep MoE, decoding, context, and KV cache mentally available because the lecture later returns to the same resource trade-off: capacity, compute, memory, and usefulness.

LinkBack: 05:00-10:00, recap of sparse MoE, decoding strategies, temperature, context, and inference optimization.

003 · Pretraining objective: broad data, next-token loss

10:00-16:00

The first major training stage is pretraining. The speaker defines it as the most expensive part of LLM development because it consumes huge corpora, large models, and large hardware budgets. The data is intentionally broad: English text, other languages, code, Wikipedia-like knowledge sources, Common Crawl-like web text, StackOverflow-like code discussions, and essentially any written distribution the model should learn. The objective stays simple: given a prefix, predict the next token. In a decoder-only text-to-text model, the training sequence begins with a beginning-of-sentence token, passes through the transformer, produces a distribution over the vocabulary at each position, and pays loss when the predicted next token differs from the corpus token. This simple objective scales because it creates labels from raw text itself. Pretraining turns raw text into supervision by shifting the sequence: every observed token becomes the answer to the prefix before it.

CoreFocus: The objective is not task success or helpfulness yet; it is distribution learning through next-token prediction.

Concept: Self-supervision - supervision created from the structure of the data itself, here by using the next token in text as the target label. Pretraining overview slide showing data mixtures and next-token objective

Notes: The key distinction for later SFT is where the loss applies: pretraining can train across the whole text stream, while instruction tuning conditions on an input and trains primarily on the response.

LinkBack: 10:00-16:00, pretraining definition, data mixture, decoder-only objective, and loss formula.

004 · Scaling laws: tokens, parameters, and compute budget

16:00-25:00

After defining the objective, the lecture asks how expensive pretraining becomes. The first compute measure is FLOPs as a count of floating-point operations. For dense LLMs, the training cost scales roughly with the product of the number of model parameters and the number of training tokens; MoE complicates the constant because not every parameter is active for every token. The second measure is FLOP/s, or operations per second, which describes hardware throughput. The lecture then introduces the Chinchilla-style question: if the total compute budget is fixed, how should one split the budget between more parameters and more tokens? The empirical answer is not simply bigger model. The model and data should be balanced, and older models can look undertrained if parameter count grew faster than token count. GPT-3 is used as an example: it had very large parameter count but comparatively fewer training tokens, so under the Chinchilla framing it did not spend compute in the most token-balanced way. Scaling laws convert model building from a slogan of bigger is better into an allocation problem between parameters, tokens, and hardware time.

CoreFocus: FLOPs measure work performed; FLOP/s measures how fast the hardware can perform that work.

Concept: Compute-optimal training - choosing model size and data size so a fixed compute budget produces the lowest expected loss. Scaling-law slide with model size and pretraining token counts

Notes: Architecture matters less in this part than scale allocation: by this lecture's assumption, modern LLMs are mostly decoder-only transformers, so token count and parameter count drive the main comparison.

LinkBack: 16:00-25:00, FLOPs notation, hardware throughput, Chinchilla law, and GPT-3 undertraining example.

005 · Training hardware state: weights are only one memory object

25:00-32:00

The lecture then moves from compute counts to the practical question of how large training actually fits on hardware. LLM training is dominated by matrix multiplications, so GPUs are the standard accelerator outside Google-specific TPU systems. But a training step must store more than the raw model weights. It must store activations from the forward pass, gradients from the backward pass, and optimizer states such as momentum-like variables for Adam-style optimizers. Each of these objects consumes GPU memory, and the speaker uses an H100 example with about 80 GB of GPU memory to show the mismatch between modern model state and per-device capacity. The learner should separate training from inference here: inference can sometimes stream a model through a different memory profile, but training must preserve enough information to update weights. The training bottleneck appears because the optimizer must remember the path to change the weights, not merely hold the weights themselves.

CoreFocus: A parameter-efficient mental model is incomplete unless it includes activations, gradients, and optimizer memory.

Concept: Optimizer state - auxiliary quantities stored by the optimizer, such as Adam moments, that guide future weight updates and increase memory use. Backward-pass slide showing gradients needed for weight updates

Notes: Memory pressure arrives before any exotic architecture issue: the same GPU that performs matrix multiplication must also store many training-time tensors.

LinkBack: 25:00-32:00, GPUs/TPUs, parameters, activations, gradients, optimizer states, and GPU memory limits.

006 · Parallelism: split data, state, or model computation

32:00-40:00

Once a single GPU cannot comfortably hold or process the training workload, the lecture introduces distribution strategies. Data parallelism divides the batch across GPUs. In the simplest form, each GPU keeps a model copy, computes a forward and backward pass on its shard, and synchronizes gradients. That improves throughput, but it repeats model state across devices. ZeRO-style variants attack that replication by sharding optimizer states, gradients, and eventually parameters across devices; as the ZeRO stage increases, memory per GPU falls, but communication and implementation complexity rise. Model parallelism splits the model-side computation itself. Expert parallelism places different MoE experts on different devices. Tensor parallelism cuts large matrix multiplications across devices. Pipeline, sequence, and context parallelism split other dimensions of the computation. Distributed training is not one trick; it is a choice about which object to partition: examples, optimizer state, parameters, tensor operations, layers, experts, or sequence context.

CoreFocus: Data parallelism asks different GPUs to process different examples; model parallelism asks different GPUs to hold or compute different parts of the model.

Concept: ZeRO - a family of data-parallel memory optimizations that shard training states so each device stores less replicated information. Model parallelism slide listing tensor, pipeline, sequence, context, and expert variants

Notes: Choose the parallelism axis from the bottleneck: batch throughput, duplicated optimizer state, large matrix multiplication, layer depth, expert placement, or long context.

LinkBack: 32:00-40:00, data parallelism, ZeRO 1/2/3, and model-parallel variants.

007 · FlashAttention setup: exact attention is memory-bound

40:00-46:00

The next optimization narrows from whole-training distribution to the attention kernel. The speaker reminds the class that self-attention computes softmax of the scaled query-key product multiplied by values. In matrix form, Q, K, and V have sequence-length rows, and the intermediate attention score matrix becomes large when context length grows. The vanilla implementation moves these matrices through the GPU's large but slower high-bandwidth memory. The hardware, however, also has much faster on-chip SRAM with far less capacity. FlashAttention exploits this hierarchy. It does not approximate attention, prune tokens, or change the formula; it changes how the exact formula is scheduled so smaller tiles move through faster memory. FlashAttention starts from a hardware fact: exact attention can be slow because moving intermediate matrices through memory costs more than the arithmetic suggests.

CoreFocus: The word exact matters: the optimization changes memory traffic and tiling, not the mathematical attention result.

Concept: Memory-bound computation - a computation whose runtime is limited more by moving data between memory levels than by raw arithmetic throughput. Self-attention formula slide motivating FlashAttention

Notes: Do not confuse FlashAttention with approximate sparse attention; the lecture presents it as an implementation-level rearrangement of standard attention.

LinkBack: 40:00-46:00, HBM/SRAM hierarchy, attention matrix formula, and vanilla memory movement.

008 · FlashAttention mechanism: tiling, scaling, and recomputation

46:00-55:00

FlashAttention works by slicing the attention computation into blocks that fit the fast memory hierarchy. Instead of materializing the full attention matrix in slow memory, it loads blocks of Q, K, and V, computes the corresponding partial softmax information, and updates the output block with the correct scaling. The transcript notes that the paper contains formulas for the scaling factor, but the lecture emphasizes the intuition rather than requiring memorization. The same theme appears again in the backward pass. Standard backpropagation stores many activations from the forward pass so they can be reused later. FlashAttention discards some of those activations and recomputes them in the backward pass. That sounds like more arithmetic, but the measured result can still be faster because high-bandwidth-memory reads and writes fall dramatically. The example compares about 40.3 memory transactions in the standard path with about 4 in the optimized path, while runtime also decreases. FlashAttention wins because it spends extra cheap computation to avoid expensive memory movement, so exact attention can use less memory and less time simultaneously.

CoreFocus: The surprising trade-off is recomputation for speed: more operations can be faster when memory traffic is the true bottleneck.

Concept: Recomputation - discarding selected forward-pass activations and recalculating them during the backward pass to reduce stored memory. FlashAttention recomputation slide comparing memory reads and writes

Notes: The study anchor is the direction of the trade: fewer HBM reads/writes dominate the added arithmetic cost.

LinkBack: 46:00-55:00, FlashAttention tiling, scaling factor intuition, recomputation, and runtime/memory comparison.

009 · Mixed precision: train faster by lowering numeric granularity carefully

55:00-62:00

The lecture then returns to the GPU specification and points out that compute throughput depends strongly on numeric precision. FP64 has high precision but low throughput; lower-precision formats such as FP32, FP16, BF16, or quantized representations can increase throughput and reduce memory. Mixed precision training uses this fact without throwing away all numerical stability. The model may keep master weights or weight updates in FP32 while performing much of the forward and backward computation in FP16. The intuition is that an individual minibatch update is noisy anyway, so the forward/backward activations and gradients often do not need every decimal place. The accumulated model weights, however, should avoid compounding quantization error over many updates. The speaker also notes that optimal precision choices vary by setup, and model builders often run smaller experiments to extrapolate scale, just as they may reproduce scaling-law relationships for their own architecture and data. Mixed precision is a controlled numerical compromise: lower precision accelerates noisy computation, while higher precision protects the accumulated model state.

CoreFocus: Precision is a training-system hyperparameter, not a universal constant; the right choice depends on hardware, model, and stability requirements.

Concept: Master weights - high-precision copies of trainable weights kept to preserve stable long-run updates while lower-precision operations run faster. Mixed precision training slide showing FP16 passes and FP32 weight update

Notes: The lecture's precision examples connect two benefits: fewer bits reduce memory footprint, and specialized hardware often executes lower-precision arithmetic faster.

LinkBack: 55:00-62:00, GPU precision throughput, FP32/FP16 mixed precision, setup-specific precision choices, and quantization preview.

010 · From pretrained language model to helpful assistant

62:00-69:00

Shervin takes over after the pretraining and optimization section by reframing what pretraining has achieved. A pretrained model has learned broad facts about language and code from huge raw corpora, but that does not mean it behaves like an assistant. The teddy-bear washer example makes the gap concrete. If the user asks whether a loved teddy bear can go in the washer, a purely pretrained next-token model may continue the text with material descriptions or likely related words instead of asking clarifying questions or giving helpful instructions. The issue is not that pretraining failed; it optimized a different target. Fine-tuning, especially supervised fine-tuning, starts from pretrained weights and trains on additional input-output pairs so the model responds to user-conditioned tasks. Pretraining teaches the model to continue language; fine-tuning teaches it which continuation counts as helpful for a user.

CoreFocus: A model can know language patterns and still fail the product behavior expected from an assistant.

Concept: Behavioral adaptation - changing model responses toward task or assistant behavior without discarding the broad representation learned during pretraining. Pretrained model behavior slide using the teddy-bear washer example

Notes: The teddy-bear example is a diagnostic: useful answering requires conditioning on user intent, not merely sampling likely text from nearby corpus patterns.

LinkBack: 62:00-69:00, transition from pretraining to fine-tuning and SFT definition.

011 · SFT objective: condition on instruction, train on answer

69:00-75:00

SFT is supervised because the training data contains labeled pairs: an input prompt and a target output. It is fine-tuning because the training starts from already trained weights and refines them on additional data. The objective can still be next-token prediction, but the location of the loss changes. In pretraining, the model learns across raw text streams from the beginning. In SFT, the user instruction is conditioning context. The model should not be rewarded for parroting the instruction; it should learn to produce the desired response after the instruction. The speaker emphasizes that the loss calculation starts on the output side. Instruction tuning is the general-assistant version of this setup: examples teach the model how to respond to story-writing, poem creation, list generation, explanation, coding, math, proof-style reasoning, and other instruction types. SFT preserves the language-model objective but masks the role of the prompt: the input sets context, and the answer carries the training signal.

CoreFocus: The same next-token machinery now serves a different conditional distribution: useful answer tokens given a user instruction.

Concept: Loss masking - excluding prompt tokens from the loss so training pressure falls on the desired response rather than on copying the input. Supervised fine-tuning slide showing loss on the output region

Notes: Remember the yellow answer-region slide: SFT uses the instruction as fixed context and fits the model on the response that should follow.

LinkBack: 69:00-75:00, SFT input-output pairs, instruction tuning, and loss placement.

012 · Instruction data mixture, safety, and generalization

75:00-84:00

The lecture expands the SFT dataset from simple instruction-answer pairs into a data mixture. Early instruction datasets were heavily human-written: people collected prompts and expert writers produced fluent, helpful answers. Modern workflows may ask strong LLMs to generate candidate outputs and then use humans or other models to review quality. The mixture covers ordinary assistant categories such as dialogue, story writing, poem generation, list production, explanation, math, proof reasoning, and code. It also includes safety behavior. A released assistant should be helpful but also harmless, so some data teaches refusal for harmful prompts and hedging for uncertain or risky claims. The speaker then uses a student question about ambiguous story writing to explain generalization. SFT examples do not need to enumerate every possible prompt; they need enough coverage in the prompt distribution that the pretrained model's broader knowledge can interpolate to new attributes. Instruction tuning works when curated examples point the pretrained distribution toward the behaviors users will actually request.

CoreFocus: Data mixture is not just volume; it is coverage of the behavioral distribution the product wants at inference time.

Concept: Instruction distribution - the range of prompts, tasks, styles, and safety cases the model is expected to handle after tuning. Instruction-tuning data categories including assistant dialogues and safety

Notes: Safety examples belong in the same training logic as helpfulness examples because both shape the conditional response distribution.

LinkBack: 75:00-84:00, SFT data mixture, generated data review, safety refusals, hedging, and prompt-distribution generalization.

013 · SFT challenges: data quality, distribution shift, and evaluation

84:00-91:00

After presenting SFT, the lecture names its main difficulties. High-quality data is expensive because it often requires humans in the loop and careful enforcement of desired behavioral rules. Data can be reused and extended over time, but curation remains resource-intensive. The second challenge is distribution alignment: the SFT data distribution must resemble the inference-time queries where the model will be judged. If users ask for a story about a specific type of poetry, the model can generalize if the SFT examples and pretraining knowledge provide enough neighboring structure, but sparse or misplaced examples can leave gaps. The speaker then introduces the evaluation problem. Helpfulness is subjective, and putting one number on it is hard. Benchmarks help, but once benchmarks become known, training data can drift toward benchmark-like examples, so high scores do not always mean real user value. SFT quality depends on three linked distributions: the curated training data, the model's pretrained knowledge, and the prompts users actually send.

CoreFocus: Generalization improves when examples cover meaningful regions of task space rather than repeating the same narrow pattern.

Concept: Evaluation distribution - the task and prompt population used to score a model, which may diverge from real deployment use. Benchmark slide listing reasoning, coding, factuality, and other evaluation categories

Notes: A benchmark number is evidence about a sampled task set, not an intrinsic measure of the model's value for every user.

LinkBack: 84:00-91:00, SFT data quality, inference-distribution mismatch, benchmark categories, and score limitations.

014 · Preference evaluation: Chatbot Arena and its biases

91:00-96:24

Because ordinary benchmark scores can miss user value, the lecture introduces Chatbot Arena-style pairwise evaluation. Users submit prompts, receive two anonymous model answers, and choose the better response; the platform aggregates pairwise comparisons into a model ranking. This turns subjective preference into a number, but it also imports the weaknesses of the preference collection process. New models can have noisy early comparisons that influence ranking. A paper is cited as showing that the leaderboard can be manipulated if a model identifies its opponent from simple prompts such as asking who it is. Human preference also does not equal factual accuracy. A user may prefer an actionable but wrong teddy-bear washing answer if they lack domain knowledge. Voter population matters too: a broad consumer group may prefer emojis or a conversational style that domain experts dislike. Safety adds another bias because users often prefer a model that answers directly even when the intended product policy should refuse. Preference rankings measure a population's comparative judgments, not a universal truth about factuality, safety, or domain fitness.

CoreFocus: Arena-style evaluation is useful because it samples lived user preference, but that same user preference can reward style, over-answering, or false confidence.

Concept: Pairwise preference ranking - an evaluation method that infers model ordering from many comparisons between two model responses to the same prompt. Preference evaluation slide illustrating pairwise model comparison

Notes: For deployment, ask which user population and which domain risk the ranking represents before treating a leaderboard position as decisive.

LinkBack: 91:00-96:24, Chatbot Arena, leaderboard noise, adversarial ranking, factuality, emoji/style preference, and safety-response bias.

015 · Alignment and mid-training: locating the post-pretraining stages

96:24-97:42

The lecture closes the fine-tuning discussion by placing SFT inside the broader alignment pipeline. The speaker says the next lecture will cover preference tuning, a further step that aligns the model with what users or product designers want. In this framing, fine-tuning plus preference tuning are called alignment because they move the pretrained language model toward desired behavior. The speaker also inserts an emerging stage called mid-training. Mid-training sits after pretraining and before fine-tuning. It keeps the same pretraining-style objective but uses data closer to the downstream tasks of interest, so the model shifts its broad distribution toward a more relevant domain before SFT teaches instruction behavior. Alignment names the behavioral stages after pretraining, while mid-training names an intermediate distribution-shaping stage before supervised instruction behavior.

CoreFocus: Keep the lifecycle order straight: pretraining, optional mid-training, SFT/instruction tuning, then preference tuning.

Concept: Mid-training - a post-pretraining, pre-SFT stage that continues next-token learning on data closer to target tasks or domains. Lifecycle slide showing pretraining, fine-tuning, and preference tuning

Notes: The lecture treats alignment as behavior shaping, not as a single algorithm.

LinkBack: 96:24-97:42, alignment definition, preference tuning preview, and mid-training note.

016 · LoRA motivation and low-rank update

97:42-102:22

The final technical block addresses the expense of fine-tuning. Full fine-tuning updates the whole weight matrix, which repeats much of the memory and compute burden of training. LoRA proposes a cheaper decomposition. Instead of changing the pretrained weight matrix directly, it freezes the base matrix W0 and learns an additive update represented as the product B A. The rank r of the bottleneck dimension is small relative to the input and output dimensions, so the number of trainable parameters falls sharply. During a forward pass, the model computes the frozen base transformation and the low-rank task-specific transformation, then adds them. The pretrained model supplies broad language competence; the low-rank update encodes the task adaptation, such as spam detection or sentiment extraction. LoRA saves fine-tuning cost by turning one large trainable matrix update into a small trainable low-rank adapter added to frozen pretrained weights.

CoreFocus: The update is parameter-efficient because r is much smaller than the original matrix dimensions.

Concept: Low-rank adapter - a pair of smaller matrices whose product approximates a task-specific update to a larger frozen weight matrix. LoRA slide showing frozen pretrained weights and low-rank matrices A and B

Notes: The formula to retain is W = W0 + B A: freeze W0, learn A and B, and keep the task-specific delta separate from the base model.

LinkBack: 97:42-102:22, LoRA motivation, frozen W0, low-rank matrices, and task-specific adapters.

017 · Where LoRA is inserted and why rank matters

102:22-105:20

After giving the basic formula, the lecture discusses where LoRA adapters are learned. The original LoRA paper focused on attention matrices, but later empirical work suggests that feed-forward blocks can be especially beneficial insertion points, and today both attention and feed-forward components may be used. This mirrors an earlier systems lesson from MoE: not every sublayer contributes equally to adaptation or cost, so placement matters. The rank r also matters. A larger rank gives the update more expressive capacity, but it increases trainable parameters. The lecture reports that performance is not always highly sensitive to r across all settings; a large part of the gain comes from the initial reduction in trainable parameters, while the exact rank becomes a design hyperparameter. LoRA is efficient because placement and rank let the engineer spend adaptation capacity where it changes behavior most per trainable parameter.

CoreFocus: Do not read LoRA as only an attention trick; modern practice may attach adapters to attention and feed-forward blocks.

Concept: Adapter placement - the choice of which model matrices receive trainable low-rank updates during parameter-efficient fine-tuning. LoRA training dynamics slide noting empirical rank and batch-size behavior

Notes: Rank is a capacity knob: too small may underfit the task delta, while too large spends more memory and compute without guaranteed benefit.

LinkBack: 102:22-105:20, attention versus feed-forward insertion and rank discussion.

018 · QLoRA: quantize the frozen base, train adapters precisely

105:20-107:23

The lecture ends by combining LoRA with quantization. QLoRA keeps the LoRA idea: the pretrained base weights are frozen, and the trainable A and B matrices carry the fine-tuning update. The extra memory reduction comes from quantizing the frozen base matrix W0. The cited method uses NF4, or 4-bit NormalFloat, which assumes the pretrained weights are approximately normally distributed and chooses quantile-based cutoffs instead of equal-width buckets. That allocates code points so roughly similar numbers of weight values fall into each quantization region, using the limited bits more efficiently than a uniform grid when the distribution is normal-like. The method also uses double quantization: after quantizing the weights, it quantizes the constants used by the quantizer, adding extra savings. The lecture notes that the primary reduction comes from the quantized base weights, while double quantization provides smaller additional benefit. QLoRA keeps adaptation quality by training the low-rank adapters in higher precision while compressing the frozen base weights that dominate VRAM.

CoreFocus: Quantize what is large and frozen; keep the small trainable update precise enough to learn the task.

Concept: NF4 - a 4-bit quantization format that uses normal-distribution quantiles to assign code points efficiently for normally distributed weights. QLoRA NF4 slide comparing uniform cutoffs with normal quantiles

Notes: The practical result is large VRAM savings for fine-tuning because the frozen base carries most of the memory mass, while A and B remain small.

LinkBack: 105:20-107:23, QLoRA, NF4, BF16 adapters, double quantization, and 16x VRAM-savings statement.

5. Suggested Answers

Question 1. Why does the lecture separate pretraining from fine-tuning instead of treating all LLM training as one next-token prediction run?

Answer 1. Pretraining and fine-tuning can both use next-token prediction machinery, but they optimize different conditional behaviors. Pretraining learns the broad text distribution from raw corpora, while fine-tuning starts from that distribution and redirects the model toward user-conditioned tasks.

Question 2. What does the pretraining objective actually optimize in a decoder-only LLM, and where does the loss get applied?

Answer 2. In pretraining, the decoder sees a token prefix and predicts the next corpus token at each position. The loss applies across the raw sequence because every next token supplies a self-supervised label.

Question 3. Why are FLOPs and FLOP/s different quantities, and why does the lecture need both when discussing LLM training cost?

Answer 3. FLOPs count the total arithmetic work required by training, while FLOP/s measures how quickly a hardware device can execute that work. The lecture needs both because model cost depends on operations demanded and hardware time depends on operations delivered per second.

Question 4. What does the Chinchilla-style scaling argument say about the trade-off between parameter count and token count?

Answer 4. For a fixed compute budget, the best model is not necessarily the largest possible parameter count. Compute-optimal training balances model size with enough training tokens so capacity and data scale together.

Question 5. Why can a model like GPT-3 be described as undertrained under compute-optimal scaling logic?

Answer 5. GPT-3 had a very large parameter count relative to its cited training-token count. Under the Chinchilla framing, a model can be undertrained when it has more parameters than its token budget can use efficiently.

Question 6. Why does training require storing more than the model weights, and why does that immediately create a GPU-memory bottleneck?

Answer 6. Training must preserve activations for backpropagation, gradients for weight updates, and optimizer states for methods such as Adam. GPU memory becomes the bottleneck because updating weights requires storing the training process, not only the final parameter tensor.

Question 7. How does data parallelism differ from model parallelism in the way it splits work across GPUs?

Answer 7. Data parallelism splits the batch across devices, usually with each device processing different examples. Model parallelism splits the model-side computation or parameters, such as experts, tensors, layers, or sequence/context dimensions.

Question 8. What problem do ZeRO-style methods solve inside data parallel training?

Answer 8. Basic data parallelism duplicates model state and optimizer state on every GPU. ZeRO reduces per-device memory by sharding optimizer states, gradients, and parameters across the data-parallel group.

Question 9. Why does FlashAttention speed up exact attention without changing the mathematical attention result?

Answer 9. FlashAttention keeps the same scaled dot-product attention formula but schedules it in blocks that fit faster memory. It improves exact attention by reducing slow HBM traffic rather than by approximating or pruning the attention computation.

Question 10. Why can FlashAttention recompute activations and still run faster than the vanilla implementation?

Answer 10. Recomputing increases arithmetic, but vanilla attention spends heavily on reads and writes of large intermediate matrices. When memory movement is more expensive than extra arithmetic, recomputation can reduce runtime as well as memory.

Question 11. Why does mixed precision keep some quantities in high precision while moving other operations to lower precision?

Answer 11. Lower precision improves memory footprint and throughput, but accumulated weights need stability over many updates. Mixed precision uses low precision where minibatch noise tolerates it and high precision where quantization error would accumulate.

Question 12. Why is a pretrained next-token model not automatically a helpful assistant in the teddy-bear washer example?

Answer 12. The pretrained model has learned likely continuations from broad text, not a product policy for helping a user solve a practical problem. It may continue with related teddy-bear material instead of giving the helpful, intent-conditioned answer the user expects.

Question 13. In SFT, why does the loss start on the answer side rather than on the user instruction itself?

Answer 13. The user instruction is the condition supplied to the model, not the text the model should learn to reproduce. SFT trains the model to generate the desired response after the prompt, so the loss belongs on the answer tokens.

Question 14. What kinds of data enter instruction tuning, and why does safety data belong in the same behavioral training story?

Answer 14. Instruction tuning uses examples of assistant dialogue, writing, lists, explanations, math, proofs, code, and other user tasks. Safety data belongs there because refusing harmful prompts and hedging uncertain claims are also learned response behaviors.

Question 15. Why can benchmark scores and Chatbot Arena rankings fail to capture the value of a model for a specific user or domain?

Answer 15. Benchmarks sample narrow task distributions, and preference rankings sample particular voter populations and styles. Neither number alone tells whether the model is factual, safe, or valuable for a specific deployment domain.

Question 16. How does the lecture define alignment, and where does mid-training fit relative to pretraining and fine-tuning?

Answer 16. The lecture frames alignment as the combination of fine-tuning and later preference tuning after pretraining. Mid-training sits between pretraining and fine-tuning, using the pretraining objective on more task-relevant data.

Question 17. What is the low-rank update in LoRA, and why does freezing W0 save fine-tuning cost?

Answer 17. LoRA represents the task update as B A and adds it to the frozen pretrained matrix W0. Freezing W0 saves cost because training only the small low-rank matrices updates far fewer parameters.

Question 18. Why does QLoRA quantize the frozen base weights but keep the trainable LoRA matrices in higher precision?

Answer 18. The frozen base weights dominate memory, while the LoRA matrices are small and carry the learning signal. QLoRA compresses the large frozen part for VRAM savings and preserves higher precision where adaptation is being learned.