Correlating a Log Line to One Task in a Large Array Job
The outcome is a run on data larger than one machine, which in practice means many parallel tasks. Diagnosing such a run requires isolating one task's story from the merged stream; without this the learner can see that something failed but not which input caused it, so they cannot re-run the failing slice. It is distinct from structured logging, which gives you fields but not identity.
This Concept is waiting for its first lesson!
The outcome is a run on data larger than one machine, which in practice means many parallel tasks. Diagnosing such a run requires isolating one task's story from the merged stream; without this the learner can see that something failed but not which input caused it, so they cannot re-run the failing slice. It is distinct from structured logging, which gives you fields but not identity.
Are you a teacher? Sign in to start contributing.
Sign In