Staleness and Resume: Re-Running Only the Steps Whose Inputs Actually Changed
Cost predictability is part of this outcome, and a multi-day pipeline that must restart from step one after a step-nine failure violates the cost and completion clauses at once. Re-running an unchanged twenty-hour step on rented compute is money burned for no new information. This is a distinct concept from declaring the steps, and it is what makes iterating on a large analysis affordable at all.
This Concept is waiting for its first lesson!
Cost predictability is part of this outcome, and a multi-day pipeline that must restart from step one after a step-nine failure violates the cost and completion clauses at once. Re-running an unchanged twenty-hour step on rented compute is money burned for no new information. This is a distinct concept from declaring the steps, and it is what makes iterating on a large analysis affordable at all.
Are you a teacher? Sign in to start contributing.
Sign In