Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Authors: Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze, Maria Lomeli, Ilze Amanda Auzina, Justin Carpentier, Gabriel Synnaeve, Hervé Jégou
Paper: https://arxiv.org/abs/2607.07386
Code: https://github.com/facebookresearch/sparse-delta-memory
Model: N/A
TL;DR
WHAT was done? The authors introduce Sparse Delta Memory (SDM), a novel architecture that dramatically expands the recurrent state capacity of gated linear RNNs by replacing dense matrix updates with a sparse read/write addressing scheme. SDM scales the hidden state of the model by up to three orders of magnitude while operating under a strict isoFLOP and constant-parameter constraint relative to dense baselines.
WHY it matters? Standard Transformers excel at long-context retrieval but are fundamentally bottlenecked by quadratic, unbounded Key-Value (KV) cache growth. Conversely, Linear RNNs and State Space Models maintain constant per-token compute but suffer from restricted state capacities that severely limit recall. SDM reconciles this trade-off, enabling constant-compute inference over sequence lengths up to 1 million tokens while significantly outperforming dense baselines on long-context benchmarks.
Details
The Recall Capacity Bottleneck in Linear Recurrent Architectures
Modern sequence modeling is defined by a fundamental tension between the expressivity of Transformers and the computational efficiency of recurrent models. While standard self-attention architectures maintain high recall over long sequences, their Key-Value (KV) caches grow linearly with sequence length, creating prohibitive memory and compute bottlenecks in agentic, long-document, or video-processing pipelines. Recurrent neural networks (RNNs) and state space models like Mamba2 [review] or xLSTM [review] compress past context into a fixed-size state, allowing constant per-token computational complexity. However, because this dense hidden state cannot scale without linearly inflating the floating-point operations (FLOPs) per token, these architectures fall short on long-context retrieval and in-context learning tasks. This limitation stems from the fact that dense state updates become computationally intractable as the state dimension increases, creating a structural barrier that has historically kept RNNs from rivaling full-attention Transformers in complex reasoning environments.
