Run a Scientific Workflow on AWS That Anyone Can Reproduce
Moving a compute-heavy analysis off one machine and onto rented cloud compute, as a working discipline. It starts with what you are actually buying when you rent a machine by the second — cores, memory, local disk and network, sold at a price that changes if you accept that the machine can be taken back — and with where a terabyte lives once it is too big for a laptop: a flat object namespace with per-request economics, a block device one machine owns, or a filesystem many machines mount at once. From there it builds the run itself: a container image as the unit of execution, a queue that turns ten thousand samples into ten thousand tasks, the difference between work that scales out and work whose ranks must all be alive at once, and the checkpointing that turns a reclaimed machine into lost minutes instead of lost days. Then it makes the result defensible: a workflow described as a graph of steps rather than a script that must be babysat, an environment pinned so it means the same thing next year, provenance recorded from inputs and code revision through to output, and the honest limits of determinism. It ends with answering for the run — logs shipped off a machine that is about to disappear, metrics that distinguish memory-starved from slow, cost attributed to a grant, and knowing when to stop a run rather than let it finish.
Updated Aug 25, 2026
