From Languages to Information
8 minutes
How repeatedly merging the most frequent adjacent pair grows a fixed-size vocabulary of variable-length tokens directly from a corpus.