Browse Teach Curate

An Analysis as a Directed Acyclic Graph of Task Dependencies

The outcome requires the analysis to finish on data larger than one machine holds and to survive a machine being taken away. Both depend on knowing which parts of the work are independent and which have to wait. A script has no such structure to inspect; the dependency graph is the representation every later node in this curation (parallel fan-out, resume, provenance) reads from. Without it the researcher can only re-run the whole thing from the top.

jeremy Video

Directed Acyclic Graphs in Data Pipeline Orchestration

A directed acyclic graph (DAG) is a graph — a structure of nodes and connections — constrained by two rules: every edge has a defined direction, and no path may return to a node it has already visite…