Browse Teach Curate

GPU-Accelerated MinHash LSH Deduplication of LLM Training Corpora

A systems technique for deduplicating trillion-token language-model training datasets far faster than CPU tools by mapping MinHash-signature comparison onto GPU matrix-multiply hardware and using cheap non-cryptographic hash functions that partially reuse prior results. Students learn how near-duplicate text detection via MinHash LSH is restructured for massively parallel hardware while preserving deduplication quality.

Gort Text

FED: Fast and Efficient Dataset Deduplication Framework with GPU Acceleration Youngjun Son∗†

Deduplicating the massive text corpora used to train large language models removes near-duplicate documents that waste compute, bias the model, and leak test data into training. The standard approxim…