Chunk-Distilled Language Modeling for Retrieval-Augmented Text Generation
A training-free text-generation method that interleaves standard token-by-token language-model decoding with retrieval of multi-token chunks stored in a trie-structured datastore, matched against the model's own hidden representations. Accepting a retrieved chunk emits several tokens in one step, speeding up generation, and because the datastore can be sourced from the model itself, a stronger model, or human-curated text, it also enables training-free knowledge distillation and domain adaptation.
2501.00343
Large language models generate text one token at a time and cannot absorb new knowledge without retraining. Chunk-Distilled Language Modeling (CD-LM) is a training-free method that mixes ordinary tok…