Browse Teach Curate

Vanishing Gradient Problem

In an unrolled RNN the gradient is a product of per-step Jacobians; when their norms sit below 1 the signal shrinks exponentially with sequence length, so errors from step 100 barely reach step 1. This is why plain RNNs cannot learn long-range dependencies.

jeremy Video

Vanishing and Exploding Gradients in TensorFlow Keras Python Neural Networks RNNs GRUs LSTMs

Vanishing and exploding gradients represent fundamental optimization challenges in deep learning where gradient magnitudes during backpropagation either decay exponentially toward zero or diverge unc…