Browse Teach Curate

Kullback-Leibler Divergence

KL(q||p) = E_q[log q - log p], the expected log-ratio measuring how one distribution diverges from another; it is nonnegative, zero only at equality, and asymmetric — the direction chosen in variational inference is what causes variance underestimation.

jeremy Video

Intuitively Understanding KL Divergence in Information Theory using Probability Distributions

Builds KL divergence from likelihood ratios: expected log-ratio between two distributions, its asymmetry, zero iff the distributions match, and its role as the fundamental measure of distributional d…