Every other tactic here assumes somebody can tell what the system is doing. That ability is not a property of good intentions or of buying a tool; it is a property of what the system emits, and what it emits is decided at design time by whoever writes the components. If a request carries no identifier across a boundary, no tool can follow it later. If the time spent is recorded only as a total, no tool can tell you which hop spent it. If a failure is logged without which caller, which resource and which attempt, the record cannot answer any question worth asking. The three kinds of output do different jobs and none replaces the others. Aggregate measurements over time tell you something is wrong and whether a target is being met. Individual event records tell you what happened in one specific case. A record of one request's path across components tells you where the time or the error actually occurred, which is the question that is hardest to answer any other way and the one that matters most once a request crosses more than two boundaries. The way to design this is to write down, before building, the questions an on-call engineer must answer during an incident, and make each one answerable. You can now state the signals a design must emit and justify each by the question it answers.
Observability is a property of a system that allows an observer to infer its internal state and understand why it is or is not functioning correctly, based solely on external outputs, in contrast to …