Stanford Online
105 minutes
How large language models are pretrained on massive web-scraped text corpora, and why the composition and cleanliness of that data matters: redundant or near-duplicate documents waste compute, bias the model toward repeated content, and can leak evaluation data into training, so corpus curation directly affects model performance and cost.