Browse Teach Curate

Tokenization

Splitting raw text into discrete units — words, subwords, or bytes — that become the model's vocabulary. Subword schemes like BPE bound vocabulary size while keeping rare words representable.

jeremy Video

Subword Tokenization Algorithms in Large Language Models

Subword tokenization is a mechanism in natural language processing that resolves the coverage gap between finite vocabulary constraints and infinite text corpora by decomposing arbitrary byte sequenc…