This concept contrasts the two families of value-estimation methods in reinforcement learning: dynamic programming, which exploits the repeated substructure of a sequential decision problem by computing the value of a state from the already-computed values of its successors across the full model, and temporal-difference (TD) learning, which performs the same bootstrapped update using single sampled transitions through the state space instead of a full sweep or a complete outcome. The defining distinction is what the update target is — a Monte-Carlo-style method updates a state's value toward the terminal outcome actually observed on that trajectory (a 0/1 or terminal return), whereas a TD method updates it toward the current estimated value of the successor state, which already aggregates every prior visit to that successor and is therefore an averaged, lower-variance target. The topic sits within reinforcement learning as the organizing taxonomy of its solution methods — dynamic programming, temporal-difference methods, and policy search — and connects the field to optimal control, operations research, and computational models of learning in psychology and neuroscience.
This concept contrasts the two families of value-estimation methods in reinforcement learning: dynamic programming, which exploits the repeated substructure of a sequential decision problem by comput…