Constant Error Carousel
The original LSTM insight: give the memory cell a self-connection of fixed weight 1.0 so the local gradient through it is exactly 1 at every step, letting error recirculate indefinitely instead of vanishing. Gates were added later precisely to control when this carousel is written, reset, and read.
This Concept is waiting for its first lesson!
The original LSTM insight: give the memory cell a self-connection of fixed weight 1.0 so the local gradient through it is exactly 1 at every step, letting error recirculate indefinitely instead of vanishing. Gates were added later precisely to control when this carousel is written, reset, and read.
Are you a teacher? Sign in to start contributing.
Sign In