Browse Teach Curate

Exploding Gradient Problem

When per-step Jacobian norms exceed 1, backpropagated gradients grow exponentially with sequence length, producing weight updates that blow past any useful region and often overflow to NaN. The multiplicative mirror image of vanishing gradients.

jeremy Video

Vanishing and Exploding Gradients in Neural Networks Using Backpropagation

The vanishing and exploding gradients phenomenon describes a class of optimization instabilities in deep learning architectures where backpropagation fails to effectively update network weights due t…