Estimated Time to Complete
Only available after login
Concepts
Transformer Architecture
Single-Node End-to-End Replication of a GPT-2-Grade Model
Perplexity as a Language Model Evaluation Metric
Instruction Tuning of Large Language Models
Autoregressive Language Modeling
Muon and Orthogonalised-Momentum Optimisers for Pretraining
Tokenizer Artefacts Behind Arithmetic and Spelling Failures
Causal Masking That Prevents a Token Attending to Its Future
Byte Pair Encoding
FlashAttention
Neural Scaling Laws
Encoder and Vision Models That Still Ship Absolute Positional Embeddings
Learned Reward Models Where No Verifier Exists
Positional Encoding and Length Extrapolation
Dense Models Where Memory Footprint Beats Parameter Count
The Unembedding Projection from Hidden State to Vocabulary Logits
Group-Relative Policy Optimization and Verifiable Rewards
Beam Search Decoding for Sequence Generation
6 of 48 Concepts are outdated
Beam search is no longer what open-ended text generation ships; maximising sequence likelihood produces bland, repetitive output, and production decoders use truncated stochastic sampling. It is still
Feedforward Networks (MLPs) in Transformers
Tokenization
Full Multi-Head Attention Where the KV Cache Is Not the Constraint
Grouped-Query and Multi-Query Attention
Rotary Position Embeddings (RoPE)
Self Attention Mechanism in Transformers
Quadratic Cost of Self-Attention in Sequence Length
Token Embeddings
Multi-Head Attention
6 of 48 Concepts are outdated
Superseded at inference by grouped-query attention: one KV head per query head makes the KV cache the binding memory cost, and grouping recovers most quality for a fraction of it.
Multi-Head Latent Attention and Low-Rank KV Compression
Temperature and Sampling in Language Model Decoding
Top-p and Top-k Truncation of the Sampling Tail
Residual Connections and Layer Normalization
Explicit Attention Matrices for Interpretability and Custom Masks
Reinforcement Learning from Human Feedback (RLHF)
6 of 48 Concepts are outdated
Superseded for routine preference alignment by direct preference optimization, which drops the separate reward model and the online RL loop for a single classification loss.
Pretraining of Large Language Models at Scale
Agent-Driven Overnight Search Over Training Code
Mixture of Experts in Deep Learning
Cross-Entropy as an Optimisation Objective
Decoding as a Memory-Bandwidth-Bound Workload
Large Language Model Pretraining and Training Corpus Quality
Greedy Decoding and Its Degeneration into Repetition
Materialising the Full Attention Matrix in High-Bandwidth Memory
6 of 48 Concepts are outdated
Superseded by FlashAttention: tiled online softmax computes identical attention without writing the N-by-N matrix to HBM, removing the real long-context bottleneck.
Softmax Function in Deep Learning
Stochastic Gradient Descent and the Adam Optimizer
Direct Preference Optimization Without a Separate Reward Model
Tokenization and the Vocabulary Trade-Off
Dense Parameter Scaling Where Every Token Activates Every Weight
6 of 48 Concepts are outdated
Superseded at the frontier by sparse mixture-of-experts: routing decouples parameter count from per-token FLOPs, so dense-only scaling is no longer the compute-efficient frontier.
KV Caching and Grouped-Query Attention
Positional Encoding
6 of 48 Concepts are outdated
Superseded by Rotary Position Embeddings: RoPE encodes relative offset inside the attention dot product and extrapolates past the trained context, which absolute schemes cannot.