Conceptual
Login

How to Build LLMs: Text Generation

Learn how to build LLMs for text generation from scratch, from tokenization and attention through pretraining to sampling and serving. You learn to size a run with neural scaling laws, turn a base model into an assistant with instruction tuning and preference optimization, and keep decoding fast.

Estimated Time to Complete

Only available after login

Concepts

Transformer Architecture Single-Node End-to-End Replication of a GPT-2-Grade Model Perplexity as a Language Model Evaluation Metric Instruction Tuning of Large Language Models Autoregressive Language Modeling Muon and Orthogonalised-Momentum Optimisers for Pretraining Tokenizer Artefacts Behind Arithmetic and Spelling Failures Causal Masking That Prevents a Token Attending to Its Future Byte Pair Encoding FlashAttention Neural Scaling Laws Encoder and Vision Models That Still Ship Absolute Positional Embeddings Learned Reward Models Where No Verifier Exists Positional Encoding and Length Extrapolation Dense Models Where Memory Footprint Beats Parameter Count The Unembedding Projection from Hidden State to Vocabulary Logits Group-Relative Policy Optimization and Verifiable Rewards Beam Search Decoding for Sequence Generation 6 of 48 Concepts are outdated Beam search is no longer what open-ended text generation ships; maximising sequence likelihood produces bland, repetitive output, and production decoders use truncated stochastic sampling. It is still Feedforward Networks (MLPs) in Transformers Tokenization Full Multi-Head Attention Where the KV Cache Is Not the Constraint Grouped-Query and Multi-Query Attention Rotary Position Embeddings (RoPE) Self Attention Mechanism in Transformers Quadratic Cost of Self-Attention in Sequence Length Token Embeddings Multi-Head Attention 6 of 48 Concepts are outdated Superseded at inference by grouped-query attention: one KV head per query head makes the KV cache the binding memory cost, and grouping recovers most quality for a fraction of it. Multi-Head Latent Attention and Low-Rank KV Compression Temperature and Sampling in Language Model Decoding Top-p and Top-k Truncation of the Sampling Tail Residual Connections and Layer Normalization Explicit Attention Matrices for Interpretability and Custom Masks Reinforcement Learning from Human Feedback (RLHF) 6 of 48 Concepts are outdated Superseded for routine preference alignment by direct preference optimization, which drops the separate reward model and the online RL loop for a single classification loss. Pretraining of Large Language Models at Scale Agent-Driven Overnight Search Over Training Code Mixture of Experts in Deep Learning Cross-Entropy as an Optimisation Objective Decoding as a Memory-Bandwidth-Bound Workload Large Language Model Pretraining and Training Corpus Quality Greedy Decoding and Its Degeneration into Repetition Materialising the Full Attention Matrix in High-Bandwidth Memory 6 of 48 Concepts are outdated Superseded by FlashAttention: tiled online softmax computes identical attention without writing the N-by-N matrix to HBM, removing the real long-context bottleneck. Softmax Function in Deep Learning Stochastic Gradient Descent and the Adam Optimizer Direct Preference Optimization Without a Separate Reward Model Tokenization and the Vocabulary Trade-Off Dense Parameter Scaling Where Every Token Activates Every Weight 6 of 48 Concepts are outdated Superseded at the frontier by sparse mixture-of-experts: routing decouples parameter count from per-token FLOPs, so dense-only scaling is no longer the compute-efficient frontier. KV Caching and Grouped-Query Attention Positional Encoding 6 of 48 Concepts are outdated Superseded by Rotary Position Embeddings: RoPE encodes relative offset inside the attention dot product and extrapolates past the trained context, which absolute schemes cannot.