About How to Build Sequence Models

Bidirectional LSTMs Transformer Architecture The Short Causal Convolution in Early Mamba Blocks The Parallel Associative Scan for Training a Linear Recurrence Recurrent Neural Networks with Gated Recurrent Units Backpropagation Gradient Clipping Autoregressive Language Modeling Hybrid Attention and State-Space Layer Stacks LSTM Output Gate Input-Dependent Selection in State Space Models LSTM Forget Gate BERT Bidirectional Transformer Encoder Pretraining for Language Representation Attention Mechanism Vanishing Gradient Problem Self-Attention Mechanism Unrolling a Recurrence into a Deep Computational Graph Stacked LSTMs Tokenization Recurrent Models under Hard Memory Ceilings on Edge Hardware Feedforward Neural Networks State-Space Models (Mamba) Quadratic Cost of Self-Attention in Sequence Length Recurrent Neural Networks Token Embeddings The Fixed-Size Context Vector Bottleneck State Space Models for Sequence Modeling in Deep Learning Recurrent Sequence-to-Sequence Translation with a Single Context Vector Backpropagation Through Time Constant Recurrent State versus a Growing Key-Value Cache Extended LSTM with Exponential Gating and Matrix Memory Sequential Dependency as the Barrier to Parallel Training State Space Models on Long Continuous Signal Streams Local Token Mixing as a Requirement of Linear Recurrent Blocks Attention-Free State Space Language Model Stacks Multiplicative Gating as a Learned Additive Path Through Time Long Short-Term Memory (LSTM) Networks Recurrent Sequence Taggers in Low-Resource and Small-Data Regimes Mamba-3 Discretisation, Complex State and MIMO Decoding Order-Dependent Prediction over Variable-Length Sequences Linear Attention and Fast-Weight Programmers in Transformers LSTM Input Gate Encoder-Decoder Architecture Structured State-Space Duality Between Linear Recurrence and Masked Attention LSTM Cell State KV Caching and Grouped-Query Attention Positional Encoding Hidden State in Recurrent Networks