Test-Time-Trained Neural Long-Term Memory for Long-Context Deep Learning
A neural memory module that keeps learning at inference time: it stores historical context by taking gradient steps on an associative-memory loss, using a surprise signal (the gradient) as momentum and a decay term as adaptive forgetting, so it trains in parallel yet updates online. Students learn how pairing this persistent long-term memory with attention (as accurate short-term memory) yields the Titans architecture family, which scales past 2M-token context windows more accurately than Transformers and linear recurrent models.
Titans: Learning to Memorize at Test Time
Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size mem…