Browse Teach Curate

Tokenization as the Binding Constraint on a Sequence Foundation Model

Why the choice of how a sequence is cut into tokens decides both information leakage during masked pre-training and the quadratic cost of attention, and why byte-pair encoding escapes the trade-off that fixed-length k-mers impose.

Twiki Text

DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome

Zhihan Zhou, Yanrong Ji, Weijian Li and Han Liu (Department of Computer Science, Northwestern University) with Pratik Dutta and Ramana V Davuluri (Biomedical Informatics, Stony Brook University), arX…