Transformer-based models & tricks

1. Summary

Meta: Source type: YouTube lecture; course: Stanford CME295 Large Language Models; lecture: Lecture 2 - Transformer-Based Models & Tricks; speaker/channel: Stanford Online; language: English; duration: about 1:47:16; transcript type: timestamped SRT; generated date: 2026-09-10.

Core: The lecture explains how the original transformer survived by changing its position encoding, normalization, attention, and training objectives, then maps those changes onto encoder-decoder, encoder-only, and decoder-only model families.

Abstract: Lecture 2 starts by repairing the mental model from Lecture 1: attention heads are parallel projections whose attention maps can reveal different token relationships. The first half then isolates three transformer components that changed after 2017: position information moved from simple input addition toward relative biases and RoPE; normalization moved from post-norm layer normalization toward pre-norm and RMSNorm; attention moved from full dense attention toward local windows and shared key-value projections. The second half uses those mechanisms to classify transformer-based architectures. T5 keeps encoder-decoder structure but trains with span corruption, BERT keeps only the encoder for bidirectional representations and classification, and modern LLMs mostly keep only the decoder because next-token prediction scales cleanly. The most important study move is to connect each trick to the pressure it relieves: length extrapolation, training stability, quadratic attention cost, KV-cache memory, task alignment, or data efficiency.

Concept: Attention head - one parallel query-key-value projection path that can learn a distinct relation pattern before all heads are concatenated.

Concept: Position embedding - information that lets attention distinguish token order even though self-attention itself is permutation-insensitive.

Concept: RoPE - rotary position embedding; a method that rotates query and key vectors so their dot product depends on relative position.

Concept: Pre-norm - a transformer block layout that normalizes the activation before the attention or feed-forward sublayer.

Concept: RMSNorm - a lighter normalization method that divides by root mean square magnitude and learns a scale parameter.

Concept: Sliding window attention - local attention that restricts each token to a neighborhood instead of the full sequence.

Concept: MQA/GQA - multi-query and grouped-query attention; methods that share key-value projections across heads to reduce KV-cache memory.

Concept: T5 - an encoder-decoder family trained by text-to-text span corruption rather than ordinary next-token prediction alone.

Concept: BERT - bidirectional encoder representations from transformers; an encoder-only architecture for contextual classification representations.

Concept: Distillation - training a smaller student model to match a larger teacher model's output distribution rather than only hard labels.

TakeHome: Position methods matter because self-attention can compare all tokens directly, but it must be told how order and distance should affect those comparisons.

TakeHome: Modern transformer tricks usually trade a small architectural change for a concrete systems gain: longer contexts, faster convergence, lower memory, or simpler scaling.

TakeHome: BERT and decoder-only LLMs are not interchangeable; BERT learns bidirectional representations for classification, while decoder-only LLMs learn causal generation.

2. Knowledge Tree

CME295 Lecture 2 - Transformer-Based Models & Tricks/
├── Lecture 1 bridge/
│   ├── self-attention formula
│   ├── query-key-value projections
│   ├── multi-head attention
│   └── attention-map interpretation
├── Position information/
│   ├── learned absolute embeddings
│   ├── sinusoidal embeddings
│   ├── relative position bias
│   ├── ALiBi linear bias
│   └── RoPE rotations
├── Normalization tricks/
│   ├── residual add and norm
│   ├── post-norm transformer
│   ├── pre-norm modern block
│   └── RMSNorm simplification
├── Attention tricks/
│   ├── O(n^2) dense attention
│   ├── sliding window attention
│   ├── receptive field across layers
│   ├── multi-query attention
│   ├── grouped-query attention
│   └── KV-cache memory pressure
├── Transformer model families/
│   ├── encoder-decoder transformer
│   ├── T5 and span corruption
│   ├── encoder-only BERT family
│   └── decoder-only LLM family
└── BERT deep dive/
    ├── bidirectional encoder representations
    ├── CLS and SEP tokens
    ├── WordPiece tokenizer
    ├── token, position, and segment embeddings
    ├── masked language modeling
    ├── next sentence prediction
    ├── fine-tuning heads
    ├── DistilBERT
    └── RoBERTa

3. Feynman Questions

Question 1. Why does the original transformer need position information if self-attention can already compare every token with every other token?

Question 2. How does the sinusoidal position encoding make similarity depend on relative distance rather than only absolute index?

Question 3. What problem do relative position bias, ALiBi, and RoPE all try to solve, and how does RoPE solve it differently?

Question 4. Why did modern transformer blocks move from post-norm layer normalization toward pre-norm and RMSNorm?

Question 5. Why does full self-attention become expensive as sequence length grows, and what does sliding window attention give up to reduce that cost?

Question 6. Why do MQA and GQA share key-value projections but usually preserve more query diversity?

Question 7. How do T5, BERT, and decoder-only LLMs differ in what part of the original transformer they keep?

Question 8. Why is BERT called bidirectional, and why does that make it useful for classification but not for open-ended generation?

Question 9. How do MLM, NSP, CLS, SEP, and segment embeddings work together in the original BERT training design?

Question 10. What do DistilBERT and RoBERTa teach about what parts of BERT were essential and what parts were optional?

4. Narrative Study Notes

001 · Recap: attention heads as parallel relation learners

00:00-00:10

The lecture opens with logistics, then immediately restores the context from Lecture 1: self-attention lets each token compare itself with all other tokens through query, key, and value vectors. The formula softmax(QK^T / sqrt(d_k))V matters here because the rest of the lecture keeps modifying the quantities around this computation rather than replacing the transformer idea. The speaker uses attention maps from the original transformer paper to answer a question about multi-head attention. A head is not a separate model with a separate MLP; it is a separate set of projection matrices that creates a separate query-key-value view of the same input. Once each head computes its own attention result, the results are concatenated and projected again. Attention maps show why this matters: one head can learn that a pronoun such as "its" points toward "law," while another can emphasize a different syntactic or semantic anchor such as "application." Multi-head attention gives the model several learned coordinate systems for asking which tokens matter, and the rest of the lecture explains how modern transformers make those coordinate systems more stable, longer-range, and cheaper.

CoreFocus: Attention heads are not decorative parallelism; they let one layer ask several relation questions at once.

Concept: Attention map - a visualization of query-key similarity weights that can show which tokens a head uses as evidence for another token.

Notes: Keep the matrix view in mind: attention heads multiply projections, compute scores, normalize them with softmax, and mix values; this is why transformer tricks often target projections, scores, or memory layout. Attention map recap from Lecture 1

LinkBack: 00:00-00:10, lecture recap and attention-head discussion.

002 · Why transformers must add position information

00:10-00:16

After the recap, the lecture starts the first major trick: position embeddings. A transformer lets tokens interact directly, so it avoids the recurrent bottleneck that processes one token after another. That strength creates a weakness. If attention only sees a bag of token embeddings, it has no built-in reason to distinguish "a cute teddy bear" from another order of the same tokens. The original transformer paper solves this by adding a position vector to each token embedding before the encoder or decoder stack. The simplest version is learned absolute position embedding: position 1 has one learnable vector, position 2 has another, and so on up to the maximum length seen in training. This method is flexible because gradient descent can learn what the training distribution rewards, but it also inherits the training distribution's boundaries. If the model learned positions only up to 512, an inference-time token at position 1,024 asks for a vector that was never trained. Absolute learned position embeddings make order available to the model, but they bind the model's sense of position to the lengths and positional patterns present in its training data.

CoreFocus: The position problem begins because self-attention removes recurrence, so order must be injected explicitly.

Concept: Learned absolute position embedding - a table of trainable vectors indexed by position and added to token embeddings.

Notes: The pressure is extrapolation: a method that works well inside the training length may fail or require interpolation outside it. Hardcoded and learned position embedding setup

LinkBack: 00:10-00:16, position embedding motivation and learned embedding limitation.

003 · Sinusoidal embeddings turn distance into dot-product structure

00:16-00:25

The second original-transformer option replaces the learned table with a deterministic formula based on sine and cosine. Each position gets a vector whose dimensions oscillate at different frequencies. Lower dimensions vary quickly, higher dimensions vary slowly, and the vector size matches the token embedding dimension so the two can be added. The lecture's key move is not to memorize the formula, but to understand why trigonometry helps. When two sinusoidal position vectors at positions m and n are dot-multiplied, terms like sin(omega m)sin(omega n) + cos(omega m)cos(omega n) combine into cos(omega(m-n)). That means the similarity between two position vectors becomes a function of their relative distance. Since embedding similarity is usually read through dot products or cosine similarity, this construction makes nearby positions more similar than distant positions near the zero-distance maximum. The method still oscillates and is not a perfect monotonic distance ruler, but it gives the transformer a structured, length-extensible position signal. Sinusoidal encoding matters because it smuggles relative distance into the same dot-product geometry that attention already uses for token similarity.

CoreFocus: Sinusoidal encodings matter when dot products turn absolute positions into recoverable relative distance.

Concept: Sinusoidal position encoding - a fixed sine-cosine vector whose dot products encode relative position through trigonometric identities.

Notes: The reason this is elegant is that the model does not need a separately trained vector for every possible future position. Trigonometric identity behind sinusoidal position encoding

LinkBack: 00:16-00:25, sine-cosine derivation and position-similarity intuition.

004 · Relative position bias moves position into the attention score

00:25-00:32

The lecture then asks whether adding position information at the input is the right place to intervene. The reason for doubt is structural: the ordering pressure is needed most inside the attention score, where QK^T decides how much one token attends to another. Modern variants therefore add position information directly inside the softmax score. T5 uses relative position bias: it bucketizes distances such as m-n, learns a bias for each bucket, and adds that bias to the attention logits. Because the softmax normalizes whatever logits it receives, the bias can raise or lower attention probability without breaking the probability sum. ALiBi takes a more deterministic route. Instead of learning the bias, it applies a linear function of relative distance, usually penalizing farther tokens more strongly. These methods share the same aim: change attention scores so closer or structurally relevant tokens are easier to attend to. Relative bias methods move order from a token-side decoration into the attention decision itself, where distance can directly raise or lower a token's chance of being used.

CoreFocus: Relative bias matters because the attention score is where distance should influence token choice.

Concept: Relative position bias - an additive term in attention logits that depends on the distance between query and key positions.

Concept: ALiBi - attention with linear bias, a deterministic distance penalty added to attention scores.

Notes: This is the first major shift from the original paper: position becomes part of score formation rather than only part of the initial representation. Linear bias in the attention layer

LinkBack: 00:25-00:32, T5 relative bias and ALiBi discussion.

005 · RoPE rotates queries and keys so attention sees relative distance

00:32-00:42

RoPE, rotary position embedding, is the lecture's main position-encoding deep dive because many modern models use it. The method returns to sine and cosine, but instead of adding a position vector to the token embedding, it rotates the query vector and the key vector by angles determined by their positions. In two dimensions, a rotation matrix uses cos(theta), -sin(theta), sin(theta), and cos(theta) to rotate a vector by angle theta. Higher-dimensional RoPE applies this idea block by block across paired dimensions, with angle schedules related to the sinusoidal frequencies from the earlier method. The important attention fact is what happens after rotation: the dot product between the rotated query at position m and the rotated key at position n can be expressed through a rotation depending on n-m. The position signal therefore enters the exact query-key comparison used by attention. The lecture also notes a long-term decay result: as relative distance grows, the upper bound of the attention weight tends to shrink, although with oscillations. RoPE is powerful because it makes the query-key dot product position-aware while preserving the attention machinery that transformers already optimize well.

CoreFocus: RoPE matters because rotation makes relative position appear inside the query-key dot product itself.

Concept: RoPE - a query-key rotation scheme that makes attention scores depend on relative distance through rotation algebra.

Notes: The useful intuition is geometric: rotate each query and key according to where it appears, then let the ordinary dot product recover a relative-position comparison. RoPE rotation of query and key vectors

LinkBack: 00:32-00:42, rotation matrix explanation and RoPE motivation.

006 · Normalization changed from post-norm to pre-norm and RMSNorm

00:42-00:51

The next transformer component is layer normalization. In the original architecture, each sublayer is wrapped by an add-and-norm operation: take the residual input, add the sublayer output, then normalize the result. Layer normalization subtracts the activation mean, divides by the activation standard deviation, and learns scale and shift parameters, often called gamma and beta. The practical purpose is training stability. Activations can become too large or too uneven across layers; normalization keeps their scale in a range where later weights can learn reliably. The lecture contrasts this with batch normalization, which normalizes across batch examples and can introduce train-inference mismatch. Modern transformer practice changes two things. First, blocks often use pre-norm: normalize before attention or the feed-forward network, then add the residual afterward. Second, many current models use RMSNorm rather than full layer norm. RMSNorm divides by the root mean square and learns only a scale, dropping mean subtraction and beta. The modern normalization trend keeps the stabilizing effect but removes unnecessary friction: normalize before the sublayer and use RMS magnitude when full layer normalization is more than the model needs.

CoreFocus: Normalization tricks matter because deep transformers train through controlled activation scale.

Concept: Post-norm - the original add-then-normalize transformer block layout.

Concept: RMSNorm - root mean square normalization, a lighter activation rescaling method with fewer learned parameters.

Notes: Normalization is not a semantic trick; it is an optimization and systems trick that makes deep transformer training less fragile. Post-norm layer normalization slide

LinkBack: 00:42-00:51, layer norm, pre-norm, RMSNorm, and batch norm comparison.

007 · Sliding window attention reduces the quadratic attention surface

00:51-00:58

The lecture then turns to attention cost. Full self-attention lets every token attend to every token, so the attention matrix has sequence length by sequence length entries. That gives O(n^2) interaction cost, which becomes expensive as context length grows. Longformer and later architectures reduce this pressure by restricting attention to a local neighborhood. In sliding window attention, each token attends only to nearby tokens within a window rather than to the full sequence. This is not merely masking a fully computed dense matrix after the fact; efficient implementations avoid materializing the whole matrix by using tiling or other structured computation. The cost reduction comes with a modeling tradeoff: one layer sees only local context. Across stacked layers, however, local neighborhoods compound. A token attends to its neighbors, those neighbors attended to their neighbors in the previous layer, and the effective receptive field expands, similar to convolutional receptive fields in computer vision. Sliding window attention buys longer contexts by replacing universal one-layer visibility with layered local communication.

CoreFocus: Sliding windows trade immediate global reach for scalable local communication across layers.

Concept: O(n^2) attention - the dense pairwise interaction cost created when every token compares with every other token.

Concept: Receptive field - the set of earlier tokens or positions that can influence a representation after one or more layers.

Notes: The design question is not whether local attention is always better; it is whether the task needs immediate global visibility or can accumulate context across layers. Sliding window attention pattern

LinkBack: 00:51-00:58, attention complexity and local attention discussion.

008 · MQA and GQA target the KV-cache bottleneck

00:58-01:03

The next attention variation does not primarily restrict which tokens can interact; it changes how projection matrices are shared across heads. Standard multi-head attention gives each head its own query, key, and value projections. Multi-query attention, MQA, keeps multiple query heads but shares one key projection and one value projection across all heads. Grouped-query attention, GQA, sits between standard MHA and MQA by sharing key-value projections within groups of heads. The lecture's systems reason is the KV cache. During autoregressive decoding, each new token attends to all previous keys and values. Those keys and values must be stored and reused, so their memory footprint grows with sequence length, layers, and heads. Sharing K and V projections reduces the cache without removing query diversity. Queries still let different heads ask different similarity questions, while shared keys and values reduce repeated storage. MQA and GQA preserve much of the multi-head questioning capacity while shrinking the key-value memory that dominates long-context decoding.

CoreFocus: MQA and GQA are memory tricks aimed at the repeated keys and values used during decoding.

Concept: KV cache - stored keys and values from previous tokens that let a decoder avoid recomputing the whole past at every generation step.

Concept: GQA - grouped-query attention, where groups of query heads share key-value projections.

Notes: This trick matters most for decoder-only generation, because generation repeatedly reuses past keys and values. Sharing attention heads with MQA

LinkBack: 00:58-01:03, MQA, GQA, standard MHA, and KV-cache motivation.

009 · Transformer families split by which blocks they keep

01:03-01:10

Shervin's part of the lecture begins by placing model families on top of the original encoder-decoder transformer. T5 keeps both encoder and decoder, but reframes every task as text-to-text. Its training objective uses span corruption: remove one or more spans from the encoder input, replace each missing span with a sentinel token, and train the decoder to reconstruct the missing spans in sequence. The decoder learns to output the first missing span, then a sentinel marker for the next missing span, and so on. This differs from plain next-token prediction because the training example is explicitly constructed around denoising missing text. T5 variants illustrate additional design axes: mT5 changes multilingual data and vocabulary, while ByT5 operates at the byte level with a much smaller vocabulary rather than a conventional 30k-token tokenizer. Encoder-decoder models like T5 keep the full transformer shape, but they change the learning problem so the model learns to map corrupted input text into reconstructed output text.

CoreFocus: T5 shows that the original encoder-decoder transformer can be repurposed through a denoising objective.

Concept: Span corruption - a denoising objective where contiguous missing text spans are replaced by sentinel tokens and reconstructed by the decoder.

Concept: Sentinel token - a special marker that identifies a missing span and organizes decoder reconstruction.

Notes: T5 remains important because it shows that transformer architecture and objective function can be separated: the same blocks support different training stories. T5 span corruption objective

LinkBack: 01:03-01:10, T5 family and span corruption explanation.

010 · Encoder-only, decoder-only, and why decoder-only won LLM scaling

01:10-01:16

The lecture then removes pieces of the original transformer. Encoder-only models drop the decoder and keep bidirectional self-attention, which makes them strong for representation and classification tasks but unable to generate autoregressively in the ordinary LLM sense. Decoder-only models drop the encoder and cross-attention; each block keeps masked self-attention and a feed-forward network. This causal mask prevents a token from attending to future tokens, so the model learns next-token prediction. Shervin emphasizes that modern LLMs are mostly decoder-only because compute was better invested in the generative side and because next-token prediction is simple, scalable, and aligned with chat-style use. Encoder-decoder architectures were popular early because the encoder seemed valuable for building a source representation, but large-scale practice shifted toward decoder-only models as next-word prediction generalized surprisingly well. The family split is architectural and objective-driven: BERT optimizes bidirectional representations, T5 optimizes text-to-text reconstruction, and modern LLMs mostly optimize causal next-token generation.

CoreFocus: The kept transformer block and training objective decide whether a model represents, reconstructs, or generates.

Concept: Decoder-only transformer - a stack of masked self-attention and feed-forward blocks trained to predict future tokens from past tokens.

Concept: Causal mask - a mask that blocks attention from a position to tokens that appear later in the sequence.

Notes: This section prevents a common confusion: transformer-based does not mean LLM-like generation; the kept block and training objective decide the behavior. BERT acronym slide introducing encoder-only models

LinkBack: 01:10-01:16, architecture families and decoder-only motivation.

011 · BERT: bidirectional encoder representations for classification

01:16-01:22

The lecture's deep dive is BERT, which stands for bidirectional encoder representations from transformers. The encoder part is straightforward: BERT takes the original transformer's encoder stack and drops the decoder. Bidirectionality is the central representational claim. Because BERT uses unmasked self-attention, each token can attend to tokens on both its left and its right. That differs from GPT-like causal models, where a token can only attend backward. The output embeddings therefore contain full-context information, making them useful for classification, token labeling, and question-answering span tasks. BERT also introduces structural tokens. CLS is placed at the beginning and becomes the pooled representation used for sequence-level classification. SEP separates sentence segments, especially for objectives involving pairs of sentences. The lecture briefly contrasts BERT with ELMo, which also built bidirectional representations but used recurrent LSTMs and therefore inherited scaling limitations from recurrence. BERT's strength is not generation; it is the ability to turn a whole bidirectional context into reusable embeddings for downstream decisions.

CoreFocus: BERT is useful when the task needs full-context embeddings rather than causal text continuation.

Concept: CLS token - a special input token whose final embedding is conventionally used for sequence-level classification.

Concept: SEP token - a separator token that marks boundaries between sentence segments.

Notes: If the task is sentiment classification, BERT can use the final CLS embedding; if the task is span extraction, it can use token-level embeddings instead. BERT encoder-only strategy slide

LinkBack: 01:16-01:22, BERT acronym, bidirectionality, CLS, SEP, and ELMo comparison.

012 · BERT input construction and MLM pretraining

01:22-01:30

BERT builds its inputs from several added representations. It uses WordPiece tokenization, a learned tokenizer that merges atomic units according to training-set likelihood and typically produces a vocabulary around tens of thousands of tokens. Each input token receives a learned token embedding, a position embedding, and a segment embedding. Segment embeddings distinguish sentence A from sentence B; all tokens in the same segment receive the same segment-type vector. This design supports BERT's original next sentence prediction setup. For masked language modeling, BERT randomly selects tokens for prediction. Most selected tokens are replaced with [MASK]; some remain unchanged; some are replaced with random words. The model must predict the original token from left and right context. This forces bidirectional contextual representations because the correct answer often depends on both preceding and following words. MLM turns unlabeled text into a supervised signal by hiding selected tokens and forcing the encoder to recover them from full bidirectional context.

CoreFocus: MLM turns raw text into a prediction task by hiding tokens and forcing bidirectional recovery.

Concept: WordPiece - a subword tokenizer that builds a vocabulary by learning useful merges from a large corpus.

Concept: Segment embedding - a learned sentence-type vector, A or B, added to every token in the corresponding segment.

Concept: Masked language modeling - a pretraining task that predicts hidden or perturbed tokens from surrounding context.

Notes: BERT's input vector is additive: token identity plus position plus segment. Its representation becomes contextual only after passing through the encoder layers. BERT input token, position, and segment embeddings

LinkBack: 01:22-01:30, WordPiece, embeddings, segment encoding, and MLM masking rules.

013 · NSP, model scale, and fine-tuning heads

01:30-01:36

BERT's second original pretraining objective is next sentence prediction, NSP. The model receives two sentences and predicts whether the second truly follows the first in the corpus or was sampled randomly. The classification head operates on the CLS embedding. The lecture also maps notation from the BERT paper: L is the number of layers, H is hidden size or embedding dimension, and A is the number of attention heads. BERT models appear in cased and uncased versions depending on preprocessing, and the original paper's scale is on the order of hundreds of millions of parameters. Fine-tuning then attaches small task-specific heads to the pretrained encoder. For sequence-level tasks such as sentiment, the classifier reads the CLS embedding. For token-level tasks such as question answering, heads predict start and end positions across token embeddings. Fine-tuning can freeze much of the pretrained model or update the full stack, depending on task and compute. BERT separates representation learning from task learning: pretraining builds contextual embeddings, and fine-tuning teaches a small head how to read them for a specific decision.

CoreFocus: Fine-tuning works because pretraining has already organized the context into readable hidden states.

Concept: Next sentence prediction - a binary pretraining task that asks whether two sentences are consecutive.

Concept: Fine-tuning head - a small task-specific network placed on top of pretrained BERT representations.

Notes: NSP was plausible as a sentence-pair signal, but later models questioned whether it was necessary. BERT hyperparameters and model notation

LinkBack: 01:30-01:36, NSP, BERT notation, model size, and fine-tuning examples.

014 · The teddy-bear example: why CLS can classify a sequence

01:36-01:42

The lecture makes BERT concrete with the sentence "This teddy bear is so cute." In an uncased setting, the text is lowercased, WordPiece tokenization converts it into vocabulary tokens, CLS is added at the beginning, SEP marks the end, and PAD tokens fill the batch length when needed. Each token vector then receives token, position, and segment embeddings before passing through the encoder. For sentiment classification, the model discards ordinary token outputs and reads the final CLS output through a classifier. That choice makes sense only because self-attention has mixed information from all tokens into the CLS representation across layers. CLS itself behaves like any other token: it has an embedding, it produces a query, key, and value, it attends to other tokens, and other tokens can attend to it. For token-level tasks, the other token outputs are not discarded; question answering can use each token embedding to predict answer start and end. CLS works as a classification handle because encoder self-attention lets that one token collect evidence from the entire bidirectional sequence.

CoreFocus: CLS can classify only because attention has mixed sequence evidence into that token across layers.

Concept: Contextual embedding - a token representation after attention has mixed information from neighboring and distant tokens.

Application: For classification, read CLS; for span extraction, score each token position; for token labeling, attach a classifier to each token embedding.

Notes: The same BERT encoder can support different output heads because the final hidden states preserve contextual information at both sequence and token levels. BERT classification example with CLS token

LinkBack: 01:36-01:42, BERT example, CLS question, and token-level output distinction.

015 · BERT limitations, distillation, and RoBERTa's simplification

01:42-01:47

The lecture closes by identifying BERT's limitations and the variants that respond to them. Early BERT context length was around 512 tokens, so longer-document tasks required approximations such as local attention to control cost. BERT-base also has roughly 110 million parameters, which creates latency and memory pressure for production classification. DistilBERT attacks that pressure through distillation. A large teacher model produces a soft probability distribution, and a smaller student model learns to match that distribution, often using KL divergence. Hinton's quoted intuition is that soft targets contain much of the model's knowledge because probabilities over wrong classes still encode similarity structure. DistilBERT reduces layers and keeps much of the performance. RoBERTa attacks a different question: are MLM and NSP both necessary? Its result suggests NSP can be removed without hurting performance, especially when training uses dynamic masking and much larger, more diverse data. The variants show that BERT's lasting contribution is bidirectional pretraining, while specific choices such as NSP, static masking, and full model size can be revised.

CoreFocus: DistilBERT and RoBERTa preserve BERT's core while revising size, masking, and NSP choices.

Concept: Soft target - a teacher model's full output distribution, which carries more information than a one-hot hard label.

Concept: Dynamic masking - changing which tokens are masked across training passes so the model sees richer prediction tasks.

Notes: DistilBERT compresses BERT; RoBERTa retrains the recipe. Both preserve the encoder-only representation idea while challenging parts of the original implementation. Distillation soft-target quote

LinkBack: 01:42-01:47, BERT limitations, DistilBERT, and RoBERTa.

5. Suggested Answers

Question 1. Why does the original transformer need position information if self-attention can already compare every token with every other token?

Answer 1. Self-attention compares token representations, but without an added position signal the same tokens in a different order can look structurally indistinguishable to the attention mechanism. Position information tells the model not only which tokens exist, but where each token sits in the sequence.

Question 2. How does the sinusoidal position encoding make similarity depend on relative distance rather than only absolute index?

Answer 2. The sine-cosine construction uses trigonometric identities so dot products between position vectors contain terms like cos(omega(m-n)). That turns the similarity of two position embeddings into a function of their relative distance.

Question 3. What problem do relative position bias, ALiBi, and RoPE all try to solve, and how does RoPE solve it differently?

Answer 3. All three methods try to make attention scores reflect how far apart tokens are, instead of leaving order only in the input embedding. RoPE solves this by rotating queries and keys so their ordinary dot product already contains relative-position information.

Question 4. Why did modern transformer blocks move from post-norm layer normalization toward pre-norm and RMSNorm?

Answer 4. The move preserves activation-scale control while improving training behavior and reducing unnecessary parameters. Pre-norm stabilizes the sublayer input, and RMSNorm keeps the useful rescaling effect with a simpler root-mean-square normalization.

Question 5. Why does full self-attention become expensive as sequence length grows, and what does sliding window attention give up to reduce that cost?

Answer 5. Full attention compares every token with every other token, so the attention surface grows as n x n. Sliding window attention reduces cost by giving up immediate global visibility and letting context spread through local windows across layers.

Question 6. Why do MQA and GQA share key-value projections but usually preserve more query diversity?

Answer 6. During decoding, keys and values from the past are stored and reused in the KV cache, so sharing them saves memory. Queries remain diverse because heads still need different ways to ask which stored information is relevant.

Question 7. How do T5, BERT, and decoder-only LLMs differ in what part of the original transformer they keep?

Answer 7. T5 keeps encoder and decoder, BERT keeps only the encoder, and modern causal LLMs mostly keep only the decoder. The kept block determines the natural task: text-to-text reconstruction, bidirectional representation, or next-token generation.

Question 8. Why is BERT called bidirectional, and why does that make it useful for classification but not for open-ended generation?

Answer 8. BERT uses unmasked encoder self-attention, so each token can attend to both left and right context. That produces strong full-context embeddings for classification, but it does not impose the causal next-token constraint needed for autoregressive generation.

Question 9. How do MLM, NSP, CLS, SEP, and segment embeddings work together in the original BERT training design?

Answer 9. MLM trains token recovery from bidirectional context; NSP trains sentence-pair classification; CLS carries the sequence-level decision; SEP marks sentence boundaries; segment embeddings tell the encoder which tokens belong to sentence A or B. Together they turn unlabeled text into supervised signals for contextual sequence and token representations.

Question 10. What do DistilBERT and RoBERTa teach about what parts of BERT were essential and what parts were optional?

Answer 10. DistilBERT shows that a smaller student can keep much of BERT's behavior by matching a teacher's soft outputs, while RoBERTa shows that NSP can be dropped when masking and data strategy improve. The essential idea is bidirectional encoder pretraining; some original training details were replaceable engineering choices.