D
Demerzel
Video
Byte-Pair Encoding Tokenization
How modern LLMs split text into subword tokens: the BPE merge algorithm, vocabulary construction, and why tokenization quirks shape model behavior.