Transformers & large language models

1. Summary

Meta: Source type: YouTube lecture; course: Stanford CME295 Large Language Models; lecture: Lecture 3 - Transformers & Large Language Models; speaker/channel: Stanford Online; language: English; duration: about 1:48:45; transcript type: timestamped SRT; generated date: 2026-09-10.

Core: The lecture connects decoder-only LLM architecture to the practical systems that make LLMs useful: sparse expert computation, token decoding, prompt/context design, and inference-time memory acceleration.

Abstract: Lecture 3 starts from Lecture 2's transformer-family taxonomy and narrows the definition of a modern LLM to a large decoder-only text-in/text-out model trained on massive token corpora with massive compute. It then explains why MoE layers let models increase total parameter capacity without activating every parameter for every token. The middle of the lecture turns the decoder's output logits into generation algorithms: greedy decoding, beam search, top-k/top-p sampling, temperature scaling, and guided decoding. The final third shifts from response quality to inference efficiency, using context length, prompt structure, in-context learning, chain-of-thought, KV caching, GQA, PagedAttention, multi-latent attention, speculative decoding, and multi-token prediction to show how modern systems trade computation, memory, latency, and robustness. The main study move is to treat LLM behavior as the combined result of architecture, probability selection, prompt context, and serving-system constraints.

Concept: Decoder-only LLM - a transformer model that keeps masked self-attention and feed-forward blocks, removes the encoder and cross-attention, and generates text by next-token prediction.

Concept: Mixture of experts - an architecture that replaces one dense subnetwork with multiple expert subnetworks and routes each token to a small selected subset.

Concept: Routing collapse - the MoE failure mode in which the router repeatedly selects only a few experts, wasting capacity and weakening specialization.

Concept: Decoding strategy - the rule that turns next-token probabilities into actual output tokens, such as greedy choice, beam search, sampling, or constrained decoding.

Concept: Temperature - a softmax scaling parameter that sharpens or flattens the next-token probability distribution before sampling.

Concept: Context length - the number of input tokens the model can attend to during generation; also called context size or window size.

Concept: In-context learning - task adaptation through examples, instructions, reasoning traces, or other prompt content supplied at inference time.

Concept: KV cache - stored key and value tensors from prior tokens, reused so autoregressive decoding does not recompute the whole prefix at every step.

Concept: PagedAttention - a memory-management method that stores KV cache in fixed-size blocks to reduce wasted reserved memory.

Concept: Speculative decoding - a serving method in which a small draft model proposes several tokens and a large target model validates them in one pass.

TakeHome: LLMs are not just big transformers; they are decoder-only language models whose usefulness depends on architecture, data scale, compute, and inference policy.

TakeHome: MoE increases model capacity by activating fewer parameters per token, but the router must be trained so expert usage does not collapse.

TakeHome: Generation quality and serving speed come from choices outside the basic transformer block: decoding, context selection, prompting, cache layout, and draft-token verification.

2. Knowledge Tree

CME295 Lecture 3 - Transformers & Large Language Models/
├── Transformer-to-LLM bridge/
│   ├── encoder-decoder, encoder-only, decoder-only taxonomy
│   ├── text-in/text-out next-token prediction
│   ├── scale by parameters, data, and compute
│   └── modern decoder-only families
├── Mixture of experts/
│   ├── expert metaphor and router gate
│   ├── dense MoE versus sparse top-k MoE
│   ├── FFN replacement inside decoder blocks
│   ├── token-level routing
│   ├── routing collapse and load-balancing loss
│   └── capacity versus active-parameter trade-off
├── Response generation/
│   ├── output logits and vocabulary softmax
│   ├── greedy decoding
│   ├── beam search and sequence scoring
│   ├── top-k and top-p sampling
│   ├── temperature scaling
│   └── guided decoding for structured outputs
├── Prompt and context behavior/
│   ├── context length, size, and window size
│   ├── context rot and distractors
│   ├── prompt structure: context, instruction, input, constraints
│   ├── zero-shot and few-shot prompting
│   ├── instruction planning
│   ├── chain of thought
│   └── self-consistency
└── Inference efficiency/
    ├── exact reuse: KV caching
    ├── attention-memory reduction: GQA and MQA
    ├── cache allocation: PagedAttention and vLLM
    ├── representation compression: multi-latent attention
    ├── approximate acceleration: speculative decoding
    └── objective-level acceleration: multi-token prediction

3. Feynman Questions

Question 1. Why does the lecture define modern LLMs as mostly decoder-only text-in/text-out models rather than simply very large BERT-like models?

Question 2. What is the MoE router doing at token level, and why does sparse MoE reduce FLOPs without reducing total parameter count?

Question 3. Why is the feed-forward network the natural place to insert MoE experts inside a decoder-only LLM?

Question 4. What causes routing collapse, and how does a load-balancing loss push the model away from it?

Question 5. Why can greedy decoding be locally optimal but bad for both sequence probability and output diversity?

Question 6. How does beam search differ from top-k or top-p sampling when each method chooses the next token?

Question 7. What does temperature do inside the softmax, and why does low temperature become spiky while high temperature becomes flatter?

Question 8. Why does guided decoding solve a different problem from sampling temperature?

Question 9. Why can longer context windows hurt retrieval if the prompt contains distractors or poorly targeted context?

Question 10. What is the trade-off between zero-shot, few-shot, chain-of-thought, and self-consistency prompting?

Question 11. Why does KV caching help autoregressive inference, and why is it mainly an inference-time technique rather than a training-time technique?

Question 12. How do GQA, PagedAttention, and multi-latent attention attack different parts of the same KV-cache memory problem?

Question 13. Why can speculative decoding preserve the target model's distribution while using a smaller draft model for speed?

Question 14. How is multi-token prediction related to speculative decoding, and what changes in the model objective?

4. Narrative Study Notes

001 · From transformer families to modern LLMs

00:00-03:40

The lecture opens by restoring the model-family map from the first two lectures. A transformer can appear as an encoder-decoder model, as in T5-style text-to-text systems; as an encoder-only model, as in BERT-style representation and classification systems; or as a decoder-only model, as in GPT-style generation systems. This taxonomy matters because the rest of the lecture uses only the third branch. In a decoder-only LLM, the encoder and cross-attention disappear, masked self-attention remains, and the model learns to continue a token sequence one token at a time. BERT can encode text well and can support classification from the [CLS] embedding, but under the lecture's current definition it is not the central LLM object because it does not generate open-ended text. The examples named in the lecture, including GPT, Llama, Gemma, DeepSeek, Mistral, and Qwen-like families, all sit in the decoder-only branch. Modern LLMs are transformer descendants, but the lecture's working object is specifically the decoder-only text generator that turns a prefix into the next-token distribution.

CoreFocus: Start from the architecture family before judging behavior: encoder-only models represent; decoder-only models generate.

Concept: Decoder-only - the transformer variant that keeps causal self-attention and feed-forward blocks while removing encoder-side machinery. Decoder-only transformer family in the lecture recap

Notes: **The lecture's later systems tricks all assume next-token prediction, so every probability, cache, prompt, and decoding method should be read as a modification around the decoder loop. **

LinkBack: 00:00-03:40, lecture recap of encoder-decoder, encoder-only, and decoder-only transformer categories.

002 · What makes the language model large

03:40-08:00

After choosing the decoder-only branch, the speaker asks what the phrase large language model means. The first answer is literal scale: modern LLMs often have billions to hundreds of billions of parameters. The second answer is data scale: pretraining uses hundreds of billions, trillions, or even tens of trillions of tokens. The third answer is compute scale: training and serving these models require large GPU resources, even though later optimizations make some inference possible on consumer hardware. The definition also has a historical component. Around 2018 or 2019, the term LLM was not yet stable, and people could sometimes group BERT-like systems under the broad language-model umbrella. In this lecture, the definition is narrower: an LLM is large in parameters, data, and compute, and it performs text-to-text generation. The word large is not one scalar; it names a three-way pressure from parameter count, token corpus size, and compute budget.

CoreFocus: An LLM is large because scale enters the weights, the data distribution, and the hardware path.

Concept: Token corpus scale - the amount of pretraining data measured in tokens rather than documents, pages, or examples. Lecture slide defining LLM scale by model size, training data, and compute

Notes: **The course treats LLM as a practical systems category, not just a neural-network size label: generation behavior depends on what the model was trained to predict and what infrastructure can run it. **

LinkBack: 03:40-08:00, LLM terminology and scale dimensions.

003 · MoE motivation: ask the right experts, not every parameter

08:00-15:30

The first architectural extension asks whether every parameter must participate in every forward pass. The lecturer uses a room metaphor: if a math question arrives in a room containing a mathematician, physicist, chemist, and historian, it is wasteful to ask everyone equally. A mixture-of-experts layer formalizes that intuition. The model contains several expert networks E_i, and a gate or router g maps the input representation to weights over those experts. In dense MoE, the model may weight all experts, so computation still touches many paths. In sparse MoE, the router selects only the top K experts, often one or two, and the output sums only the selected expert outputs. The key systems gain is that total model capacity can grow while active computation per token stays controlled. MoE separates total parameters from active parameters, so the model can become larger without forcing every token to pay for every expert.

CoreFocus: The router is the architectural price of sparsity: it decides which specialized subnetworks each token actually uses.

Concept: Sparse MoE - a mixture-of-experts design that activates only a selected top-k subset of experts for each input. Mixture of experts router and weighted expert outputs

Notes: **FLOPs enter here as the compute measure: sparse MoE lowers forward-pass operations compared with activating the full dense model, even though the stored parameter count increases. **

LinkBack: 08:00-15:30, expert metaphor, gate/router notation, dense versus sparse MoE, and FLOPs motivation.

004 · Where MoE lives: the FFN is the expensive sublayer

15:30-22:00

The lecture then places MoE inside the decoder block. The candidate locations are masked self-attention, the feed-forward neural network, and normalization. The feed-forward network is the natural target because it expands a d_model vector into a much wider d_ff space and then projects back down. That expansion gives the FFN a large parameter and operation footprint, often larger than the attention projection footprint. Attention uses query, key, and value projection matrices whose head dimensions are comparatively controlled, while the FFN width can be thousands or tens of thousands. Modern MoE LLMs therefore replace the ordinary FFN with several FFN experts and let a token-level router choose which expert path to activate. This preserves the decoder scaffold while changing the costly middle computation inside each block. MoE usually replaces the decoder FFN because that is where widening creates enough computation to make sparse expert routing worthwhile.

CoreFocus: Do not put experts everywhere; put them where the dense block spends most of its multiply-add budget.

Concept: Active parameters - the subset of model parameters actually used for a given token's forward pass. Routing collapse slide for MoE training challenge

Notes: **The expert is not an attention head. Attention heads remain one mechanism; MoE experts are usually separate FFN alternatives selected per token and per layer. **

LinkBack: 15:30-22:00, FFN parameter scale, sparse expert placement, and token-level routing.

005 · Routing collapse, load balancing, and capacity trade-offs

22:00-37:30

Once routing becomes trainable, the next problem is distribution of work. If the router always chooses the same expert, the model owns many stored experts but uses only a few, which the lecture calls routing collapse. To counter that failure, MoE training adds a load-balancing term to the loss. The exact quantities discussed include average routing probability from the gate outputs and empirical expert usage. The loss pushes selected tokens and routing probabilities toward broader expert use, so experts receive gradient signal and can specialize. The lecture also answers several practical questions: the router and experts are trained jointly; adding experts increases total parameters; sparse activation keeps active parameters and FLOPs bounded; gates are layer-specific; and routing can vary by token and by decoder layer. A Mistral-style visualization colors each token by selected expert, making the health criterion visible: not every token should collapse to the same color. The MoE trade-off is capacity for routing discipline: the model gains many expert parameters only if the gate keeps enough experts alive during training and inference.

CoreFocus: MoE succeeds when expert capacity grows faster than active compute while routing remains balanced enough to train the capacity.

Concept: Load-balancing loss - an auxiliary loss term that penalizes collapsed expert usage and encourages more even routing. Token-level expert routing visualization

Notes: **Switch Transformer is cited as a reading example for scaling to trillion-parameter MoE capacity; the important lesson is not the number itself but the distinction between total parameters and active parameters. **

LinkBack: 22:00-37:30, differentiability questions, expert count, per-layer routers, Mistral routing visualization, and MoE summary.

006 · Next-token probabilities and the greedy baseline

37:30-40:30

After MoE, the lecture moves from architecture to response generation. A decoder-only LLM maps a prefix, such as beginning-of-sentence plus previous tokens, to a probability distribution over the vocabulary. The simplest decoding rule is greedy: choose the token with maximum probability at each step. That rule looks sensible because it chooses the model's strongest immediate preference, but it has two limitations. First, the transformer computation is deterministic for a fixed prefix, so greedy decoding makes the same choice every time and produces low diversity. Second, a locally highest-probability token can lead to a poor continuation; sequence probability depends on the product, or log-sum, of all future conditional probabilities. A lower-probability first token can lead to a much better full sequence if the later path has stronger conditional probabilities. Greedy decoding optimizes the next local decision, not the whole generated sequence or the diversity of possible responses.

CoreFocus: The probability distribution is not the answer; the decoding rule turns probabilities into a particular text.

Concept: Greedy decoding - next-token selection by choosing the vocabulary item with the highest current probability. Greedy decoding chooses the highest-probability next token

Notes: **Greedy decoding is useful as a baseline because it exposes the local-versus-global issue before adding beam search or sampling. **

LinkBack: 37:30-40:30, next-token prediction setup and greedy decoding limitations.

007 · Beam search tracks several likely paths

40:30-47:30

Beam search responds to greedy decoding by keeping multiple candidate paths alive. The beam size or beam width K names how many paths the algorithm tracks. Starting from the beginning-of-sentence token, beam search expands candidate next tokens, evaluates continuations from each branch, and keeps the top K partial sequences according to accumulated score. The score is usually the sum of log probabilities, which converts a product of conditional probabilities into an additive sequence score. This is more global than greedy decoding because a branch with a lower first step can survive if its later tokens make the full sequence more likely. The lecture also names a length problem: multiplying probabilities less than one makes longer sequences receive smaller raw probabilities, so practical beam search uses length correction or normalization terms. Even with that correction, beam search still favors high-likelihood text and often lacks the diversity or creativity users expect from chat systems. Beam search broadens the search path, but it still searches for likely sequences rather than sampling a diverse distribution of plausible continuations.

CoreFocus: Beam width buys global search structure, not creativity.

Concept: Beam width - the number of partial output paths retained at each decoding step. Beam search keeps the two most likely partial paths

Notes: **Beam search is common in translation-like tasks where likely sequence quality matters more than conversational variety. **

LinkBack: 40:30-47:30, beam search path expansion, log-probability scoring, length penalty, and diversity limitation.

008 · Sampling, top-k, and top-p trade likelihood for diversity

47:30-52:30

Sampling changes the generation objective. Instead of trying to find the highest-probability path, the model draws the next token from the probability distribution. Plain sampling can theoretically choose very low-probability tokens, so practical systems restrict the candidate set first. Top-k sampling keeps only the K highest-probability tokens and samples among them. Top-p, or nucleus sampling, keeps the smallest set of high-probability tokens whose cumulative probability exceeds a threshold p, then samples inside that adaptive set. The contrast with beam search is important: beam search maintains multiple paths to approximate a high-scoring sequence, while top-k and top-p reshape the support of the distribution used for stochastic choice. These methods explain why the same prompt can produce different outputs across runs even when the underlying transformer weights and forward computations are fixed. Sampling methods make generation useful for open-ended language by preserving plausible alternatives instead of collapsing every step to a single best token.

CoreFocus: Top-k uses a fixed count; top-p uses a probability-mass threshold.

Concept: Nucleus sampling - another name for top-p sampling, where the candidate set changes size with the distribution's concentration. Top-k sampling restricts selection to the highest-probability tokens

Notes: **Sampling is not randomness for its own sake; it is a controlled way to preserve multiple plausible continuations while filtering implausible tails. **

LinkBack: 47:30-52:30, sampling motivation, top-k sampling, and top-p sampling.

009 · Temperature reshapes the softmax distribution

52:30-01:05:00

The lecture next asks where next-token probabilities come from. The decoder produces a final hidden representation, a linear layer projects it to vocabulary-sized logits, and softmax converts logits into probabilities that sum to one. Temperature enters inside this softmax as a divisor on the logits: lower temperature magnifies logit differences, while higher temperature compresses them. The lecturer derives the limiting intuition by factoring out the largest logit x_k. When temperature approaches zero, terms for all logits below x_k shrink toward zero after exponentiation, so the distribution concentrates on the maximum-logit token. When temperature grows large, logit differences divided by temperature approach zero, exponentials approach one, and the distribution approaches uniform. In practice, low temperature makes outputs more deterministic and conservative, while high temperature increases creative variation. The lecture adds a useful engineering caveat: even T = 0 may not guarantee bitwise identical outputs because GPU reduction order and hardware nondeterminism can change numerical results. Temperature does not change the transformer computation; it changes how sharply the final logits become a sampling distribution.

CoreFocus: Low temperature sharpens probability mass around the top logit; high temperature spreads mass across more tokens.

Concept: Logit - the pre-softmax score assigned to a vocabulary token before normalization into probability. Temperature changes the next-token probability distribution

Notes: **Use lower temperature for deterministic or format-sensitive tasks; use higher temperature when diverse wording, ideation, or creative variation matters. **

LinkBack: 52:30-01:05:00, vocabulary projection, softmax, temperature derivation, and nondeterminism caveat.

010 · Guided decoding constrains invalid next tokens

01:05:00-01:08:50

Guided decoding solves a different problem from temperature. Temperature asks how random or sharp sampling should be; guided decoding asks which next tokens are valid under an external output constraint. The lecture uses JSON as the example. A naive method asks the LLM to produce JSON, checks whether the result parses, and retries when it fails. Guided decoding moves the constraint into generation itself. If the output must begin with an opening brace, all other next tokens are filtered out. If a property name or punctuation mark must follow under a grammar, the decoder restricts the candidate set to tokens consistent with that grammar or finite-state machine. When more than one token remains valid, the usual sampling or selection strategy can still operate inside the valid set. Guided decoding makes structure a generation-time constraint, so the model cannot spend probability mass on tokens that would make the output invalid.

CoreFocus: Temperature shapes probability; guided decoding masks the vocabulary by validity.

Concept: Grammar-constrained decoding - generation that uses a grammar, parser state, or finite-state machine to reject invalid next tokens. Guided decoding for JSON-format output

Notes: **This is especially relevant for production systems that need JSON, tool calls, SQL-like strings, or other structured outputs where post-hoc retries waste latency. **

LinkBack: 01:05:00-01:08:50, guided decoding motivation and valid-token filtering.

011 · Context length is capacity, not guaranteed retrieval

01:08:50-01:12:30

The second instructor shifts from response generation to prompting. The first vocabulary item is context length, also called context size or window size: the number of input tokens the model can attend to. Modern models advertise very large context windows, sometimes in the million-token range, but the lecture warns that more context does not automatically solve retrieval. The cited context-rot or needle-in-a-haystack result tests whether a model can recover an answer buried inside increasingly long text. As context grows, retrieval can degrade, especially when distractors appear in the prompt. This point connects directly to attention: although the context window defines what tokens are available to self-attention, the model still has to use the right evidence. For retrieval-augmented use, the practical lesson is to target the relevant context rather than dumping everything into the prompt. A long context window expands what the model can see, but poorly selected context can still hide the answer behind distractors.

CoreFocus: Context length is an upper bound on visible tokens, not a promise that the model will retrieve the right token.

Concept: Context rot - degradation of retrieval or grounding behavior as the prompt grows longer and contains more distracting material. Context rot warning in long-context prompting

Notes: **For Sid's use, this is the systems reason to curate documents, chunks, and retrieval context instead of trusting maximum context length alone. **

LinkBack: 01:08:50-01:12:30, context length terminology, needle-in-haystack retrieval, distractors, and self-attention connection.

012 · Prompt structure supplies the model's temporary task environment

01:12:30-01:18:00

The lecture gives a practical prompt skeleton rather than a formal theory. A prompt often contains context, instructions, the actual input, and constraints. In the teddy-bear example, context describes the setting, instructions describe the desired task, input supplies the variable content, and constraints restrict the output's suitability or format. This structure matters because in-context learning happens at inference time: the model receives a temporary task environment inside the prompt rather than new weight updates. Zero-shot prompting gives only the query or instruction. Few-shot prompting adds input-output examples before the target query. Examples usually help, but they also consume context window and can over-constrain the model to a finite pattern. The lecture notes a modern nuance: stronger models can sometimes match or beat few-shot examples when instructions are clearer, more reasoning-based, or plan-oriented. A prompt is not just a question; it is a temporary data structure that tells the LLM what world, task, input, and constraints to condition on.

CoreFocus: Prompt quality depends on what context is included, what task is stated, and what irrelevant evidence is excluded.

Concept: Zero-shot prompting - asking the model to perform a task without supplying worked examples in the prompt.

Concept: Few-shot prompting - supplying example input-output pairs so the model can infer the task pattern in context. Zero-shot and few-shot in-context learning comparison

Notes: **Few-shot examples are useful when the pattern is hard to describe, but clear instructions may generalize better when the deployment distribution differs from the examples. **

LinkBack: 01:12:30-01:18:00, prompt components, zero-shot prompting, few-shot prompting, and instruction quality trade-off.

013 · Chain of thought and self-consistency expose reasoning traces

01:18:00-01:25:00

Chain-of-thought prompting asks the model to produce reasoning before the final answer. In the teddy-bear age example, the direct answer may fail because the model has to combine dates and age arithmetic. A reasoning trace gives intermediate steps, which can improve benchmark performance and make failure modes more inspectable. The lecture emphasizes the debugging value: with ordinary neural networks, one might inspect weights or activations, but with LLM applications, developers often debug token outputs. If the reasoning says the year is 2019 when the prompt intended a later year, the trace exposes a context problem. Self-consistency builds on chain of thought by sampling several independent reasoning paths in parallel, parsing their final answers, and using majority voting. The method increases robustness but costs more generated tokens and more inference work. Chain of thought buys interpretability and sometimes accuracy by making the model place its intermediate reasoning in tokens the user can inspect.

CoreFocus: Reasoning traces are useful because LLM application debugging happens at the token interface.

Concept: Self-consistency - sampling multiple reasoning traces and selecting the answer that appears most often after parsing final responses. Chain-of-thought prompting improves reasoning visibility

Notes: **The gain is not free: chain of thought and self-consistency increase output length, latency, and cost, so they should be reserved for tasks where reasoning reliability matters. **

LinkBack: 01:18:00-01:25:00, chain-of-thought rationale, debugging, and self-consistency majority voting.

014 · Inference efficiency starts with exact reuse

01:25:00-01:32:00

The final lecture segment asks how to generate many tokens efficiently from very large models. The speaker divides techniques into exact optimizations, which compute the same result more efficiently, and approximate optimizations, which change the computation while trying to preserve quality. The first exact idea is KV caching. Autoregressive decoding generates one token, appends it to the prefix, and then generates the next token. Without caching, the model would repeatedly recompute keys and values for earlier tokens even though those earlier token representations do not change. KV caching stores the key and value tensors from previous tokens and reuses them when the new token attends back to the prefix. This does not arise in the same way during training because teacher forcing processes the whole target sequence at once. KV caching accelerates inference by reusing the prefix computations that autoregressive generation would otherwise repeat at every decoding step.

CoreFocus: The cache changes serving efficiency, not the mathematical target of next-token prediction.

Concept: Teacher forcing - a training setup that feeds the known sequence tokens together, so the sequential cache-reuse problem is not the same as inference. KV caching reuses key and value tensors for previous tokens

Notes: **Read KV cache as the memory counterpart of causal attention: every new token needs old keys and values, so storing them avoids repeated projection work. **

LinkBack: 01:25:00-01:32:00, exact versus approximate efficiency, KV caching, and teacher-forcing question.

015 · GQA and PagedAttention reduce cache pressure in different ways

01:32:00-01:39:00

Once key and value tensors are cached, the next bottleneck is memory. The first response reuses Lecture 2's grouped-query and multi-query attention ideas. In full multi-head attention, each head can have its own key and value projections. GQA groups query heads so they share fewer key-value sets, and MQA pushes the sharing further. This reduces the number of cached key-value vectors without removing all query-head diversity. The second response is PagedAttention, which attacks allocation waste rather than tensor shape. A naive inference server may reserve memory for the maximum context length of each request even when most requests finish early. That creates internal fragmentation inside reserved blocks and external fragmentation across the memory allocator. PagedAttention, used in vLLM, stores KV cache in fixed-size blocks and maps token positions to blocks, so the server allocates only what a request actually grows into. GQA shrinks what must be stored per token, while PagedAttention changes how those stored tensors occupy GPU memory.

CoreFocus: Memory efficiency has two layers: reduce the cache tensor itself, then manage the remaining cache without fragmentation.

Concept: PagedAttention - a block-based KV-cache allocator that avoids reserving one large contiguous maximum-context buffer per request. PagedAttention stores KV cache in fixed-size non-contiguous blocks

Notes: **The lecture's serving picture is multi-user: wasted memory per request reduces throughput because fewer requests fit on the same hardware. **

LinkBack: 01:32:00-01:39:00, GQA/MQA cache reduction, naive allocation, internal/external fragmentation, PagedAttention, and vLLM.

016 · Multi-latent attention compresses the cached representation

01:39:00-01:43:30

DeepSeek-style multi-latent attention attacks the KV-cache problem by changing the representation stored for each token. In vanilla multi-head attention, each token representation produces separate key and value projections across heads, so the cache must hold many long vectors per transformer block. The lecture describes a compression-decompression pattern: project the token representation into a lower-dimensional latent space, store that compact latent, and then decompress it into key or value form when needed. The clever part is sharing the compression matrix across keys and values, and even across heads, while retaining different decompression matrices so the model can still recover useful key and value representations. This means each token can have one compact cached representation per block rather than many separate key-value embeddings. The DeepSeek V2 paper is also reported to find performance benefits, plausibly from a regularizing effect of shared latent representations. Multi-latent attention reduces KV-cache memory by caching a shared low-rank latent and learning how to expand it back into attention keys and values.

CoreFocus: The cache stores the bottleneck representation; decompression restores the attention-specific forms only when needed.

Concept: Low-rank cache latent - a smaller stored vector that replaces multiple larger key/value vectors and is later expanded by learned projections. Multi-latent attention after compression shares a compact latent representation

Notes: **The low-rank dimension is a fixed design choice, not a per-token adaptive variable in the lecture's description. **

LinkBack: 01:39:00-01:43:30, DeepSeek V2 multi-latent attention, compression/decompression, and shared latent cache.

017 · Speculative decoding: draft fast, verify with the target model

01:43:30-01:46:30

The approximate-efficiency section starts with speculative decoding. The method uses a smaller draft model to propose several next tokens quickly, then asks the larger target model to validate those draft tokens. The motivation is hardware-aware: at inference time, generation can be memory-bound, so running one large forward pass over several drafted positions can be more efficient than running many separate large-model passes one token at a time. The draft model proposes a sequence such as is cute and smart; the target model computes probability distributions for those positions in parallel; and an acceptance/rejection rule decides which drafted tokens can be kept. If a token is rejected, the method samples an alternative from a corrected distribution and exits the draft stretch. The mathematical detail is that the acceptance rule can preserve the target model's distribution under the assumptions of the speculative decoding paper. Speculative decoding uses the small model for cheap proposals and the large model for distribution-preserving verification.

CoreFocus: The small model does not replace the target model; it proposes tokens that the target model can accept or reject efficiently.

Concept: Draft model - a smaller, faster model used to generate candidate tokens before target-model validation. Speculative decoding acceptance and rejection sampling rule

Notes: **The important distinction is exact output distribution versus approximate work path: the computation path changes, but the accepted-token distribution is designed to match the target. **

LinkBack: 01:43:30-01:46:30, speculative decoding motivation, draft model, target model, and acceptance/rejection rule.

018 · Why the target model scores the draft block at once

01:46:30-01:47:40

The lecture spends extra time on why speculative decoding includes all draft tokens, plus the next token after the draft, in the target-model pass. A transformer forward pass over the draft block gives the target distribution for each drafted position, and it also gives the distribution for the token that follows the accepted draft stretch. That means the large model obtains several validation decisions and one next-token distribution in one memory-heavy pass. If all draft tokens are accepted, the system has advanced multiple generated tokens at the cost of one target-model evaluation. If a rejection occurs, the algorithm samples from the corrected distribution at the rejection point and resumes. This is why the technique is attractive when memory movement, not arithmetic, limits throughput. The target model evaluates a whole draft block because one pass can validate multiple candidate tokens and produce the next distribution needed after acceptance.

CoreFocus: Speculation creates speed only when block validation costs less than many separate target-model decoding steps.

Concept: Memory-bound inference - a serving regime where moving model weights and cache tensors limits speed more than raw arithmetic operations. Target LLM validates a block of draft tokens in speculative decoding

Notes: **This section links serving hardware to algorithm design: the same mathematical model can run faster when tokens are batched into one target pass. **

LinkBack: 01:46:30-01:47:40, target pass over draft tokens, rejection behavior, and memory-bound rationale.

019 · Multi-token prediction embeds the draft idea in one model

01:47:40-01:48:45

The final technique, multi-token prediction, resembles speculative decoding but changes the model itself. Instead of training only one next-token prediction head, the model attaches multiple prediction heads to the final decoder representation and trains them to predict several future tokens, such as t+1, t+2, up to t+k. At test time, those heads act like an internal draft generator, while the main first head plays the target role. Draft tokens from the auxiliary heads are fed back and accepted or rejected by the main path, often with a greedy variant because the model objective and architecture no longer give exactly the same speculative-decoding distribution proof. The lecture's closing slide maps the whole efficiency segment: attention tricks reduce memory, PagedAttention manages cache allocation, multi-latent attention compresses cached representation, speculative decoding accelerates token generation with a separate draft model, and multi-token prediction embeds drafting into training. Multi-token prediction turns draft generation into a training objective, so one model learns to propose several future tokens instead of only the immediate next token.

CoreFocus: The objective changes from next-token prediction alone to multiple future-token predictions from the same decoder state.

Concept: Multi-token prediction - a training and inference scheme in which multiple output heads predict several future tokens from one decoder representation. Multi-token prediction heads generate several tokens at once

Notes: **This ending reinforces the lecture's main systems logic: modern LLM performance depends as much on decoding and serving design as on the base transformer block. **

LinkBack: 01:47:40-01:48:45, multi-token prediction and final technique summary.

5. Suggested Answers

Question 1. Why does the lecture define modern LLMs as mostly decoder-only text-in/text-out models rather than simply very large BERT-like models?

Answer 1. BERT is large and transformer-based, but it is encoder-only and mainly produces contextual representations for tasks such as classification. Under this lecture's working definition, a modern LLM is large in parameters, data, and compute while also being a decoder-only model that generates text through next-token prediction.

Question 2. What is the MoE router doing at token level, and why does sparse MoE reduce FLOPs without reducing total parameter count?

Answer 2. The router maps each token representation to expert scores and selects a small subset of experts for that token. Sparse MoE can store many expert parameters, but only the selected experts run in a forward pass, so active FLOPs stay lower than dense all-expert computation.

Question 3. Why is the feed-forward network the natural place to insert MoE experts inside a decoder-only LLM?

Answer 3. The FFN expands d_model into a wider d_ff space and projects back, so it carries a large share of decoder-block parameters and operations. Replacing the FFN with routed FFN experts makes sparsity attack the expensive sublayer rather than a minor component.

Question 4. What causes routing collapse, and how does a load-balancing loss push the model away from it?

Answer 4. Routing collapse occurs when the gate repeatedly selects only a few experts, leaving other experts unused and poorly trained. A load-balancing loss uses routing probabilities and expert usage to penalize that pattern, so the router is pressured to distribute tokens broadly enough for experts to remain active.

Question 5. Why can greedy decoding be locally optimal but bad for both sequence probability and output diversity?

Answer 5. Greedy decoding always chooses the highest-probability next token, but the best immediate token can lead to weak later conditional probabilities. It also makes deterministic repeated choices for the same prefix, so greedy decoding can miss better full paths and collapse response diversity.

Question 6. How does beam search differ from top-k or top-p sampling when each method chooses the next token?

Answer 6. Beam search keeps several partial sequences and scores them by accumulated log probability, aiming for a more likely full sequence. Top-k and top-p first restrict the candidate token set and then sample, so beam search searches likely paths while top-k/top-p preserve stochastic variety inside a filtered distribution.

Question 7. What does temperature do inside the softmax, and why does low temperature become spiky while high temperature becomes flatter?

Answer 7. Temperature divides the logits before exponentiation. When temperature is small, logit gaps become large and non-maximum tokens shrink toward zero; when it is large, gaps shrink and probabilities equalize, so temperature controls how sharply logits become a probability distribution.

Question 8. Why does guided decoding solve a different problem from sampling temperature?

Answer 8. Temperature changes how concentrated or random the distribution is, but it does not know whether a token is valid JSON or valid grammar. Guided decoding filters invalid next tokens during generation, so it enforces structural validity before the sampling rule chooses among allowed tokens.

Question 9. Why can longer context windows hurt retrieval if the prompt contains distractors or poorly targeted context?

Answer 9. A longer context window lets the model see more tokens, but it also increases the chance that irrelevant material competes with the answer. The lecture's context-rot point is that visibility is not retrieval; targeted context matters because distractors can bury the useful evidence.

Question 10. What is the trade-off between zero-shot, few-shot, chain-of-thought, and self-consistency prompting?

Answer 10. Zero-shot is cheap and unconstrained but gives little pattern evidence; few-shot gives examples but consumes context and can overfit the prompt pattern; chain of thought adds inspectable reasoning but costs tokens; self-consistency samples multiple reasoning paths for robustness. Each method buys task guidance or reliability by spending context, latency, or output tokens.

Question 11. Why does KV caching help autoregressive inference, and why is it mainly an inference-time technique rather than a training-time technique?

Answer 11. During inference, each new token attends to the same previous prefix, so recomputing old keys and values would waste work. During teacher-forced training, the known sequence is processed together, so KV caching mainly helps the sequential decoding loop by reusing prefix key-value tensors.

Question 12. How do GQA, PagedAttention, and multi-latent attention attack different parts of the same KV-cache memory problem?

Answer 12. GQA reduces the number of key-value sets by sharing them across query heads; PagedAttention reduces allocation waste by storing cache in fixed-size blocks; multi-latent attention stores a compact shared latent and decompresses it later. They target cache shape, cache placement, and cached representation size.

Question 13. Why can speculative decoding preserve the target model's distribution while using a smaller draft model for speed?

Answer 13. The draft model only proposes candidate tokens; the target model still computes the probabilities used to accept, reject, or correct those candidates. Under the sampling rule from the paper, the large model validates the draft so the final distribution matches the target model rather than the small model alone.

Question 14. How is multi-token prediction related to speculative decoding, and what changes in the model objective?

Answer 14. Both methods try to advance several tokens faster than ordinary one-token decoding. Speculative decoding uses a separate draft model, while multi-token prediction attaches several future-token heads to the same model and trains them jointly, so drafting becomes part of the model's training objective rather than only an external serving trick.