Browse Teach Curate

Why Many Small Files Cost More Than One Large File

Scientific data arrives as directories of per-sample, per-tile or per-timestep files, so this pattern is nearly universal in the target audience and it is the single most common reason a first cloud run is both slow and expensive. The trade against random access keeps it a judgement, not a rule.

jeremy Video

Compacting Small Files to Restore Query Performance in Apache Spark

The small file problem is the degradation of distributed query performance that arises when a dataset is stored as a very large number of small objects, because the cost of object storage access is d…