State space models: the alternative to the transformer

Since the introduction of the self-attention mechanism, the transformer architecture has dominated virtually every modern language model. Yet this dominant design suffers from a fundamental computational bottleneck: quadratic complexity in time and memory as input sequences grow longer. State space models (SSMs), with variants such as S4 and the popular Mamba architecture, offer a mathematical alternative with linear scalability. In this article we look at how these models work, which technical advantages they offer for long contexts, and why hybrid forms combining attention mechanisms and state spaces are redefining the current standard.

The quadratic limit of classic transformers

To understand why state space models attract so much attention in academic circles and industrial AI laboratories, we first have to look at the computational limits of the traditional transformer. Anyone wanting to check how a transformer works and why self-attention is so powerfulwill see immediately that every token in a sequence calculates its relationship with every other preceding token. This so-called self-attention mechanism requires a computational complexity of O(N²), where N represents the length of the sequence.

With relatively short context windows of 2,048 to 4,096 tokens, the computing time needed remains manageable. As soon as we scale context windows up to tens of thousands or millions of tokens for document analysis or audio processing, however, the memory footprint explodes. The KV cache (key-value cache), which stores all earlier states during inference to avoid recalculation, grows linearly with context length and linearly with batch size. On graphics accelerators this leads to considerable memory pressure and bandwidth problems.

In the ongoing development around ever larger context windows , researchers therefore run up against physical barriers. FlashAttention and context caching partly relieve the I/O bottleneck through hardware-specific optimisations, but the underlying algebraic complexity of self-attention remains quadratic during training. State space models approach this sequential processing problem from a fundamentally different mathematical foundation.

The origin: continuous systems and the S4 architecture

State space models have their roots in classic control engineering and signal processing. In essence, a continuous linear time-invariant (LTI) system describes how a continuous input x(t) is mapped to a hidden state h(t) through differential equations, in order then to generate an output y(t) . This continuous system is formally represented through linear transformations:

h'(t) = A * h(t) + B * x(t)
y(t)  = C * h(t) + D * x(t)

To make these differential equations applicable within discrete numerical architectures on computers, the continuous matrices (A, B) are discretised into discrete parameters (Ā, B̄) using a step size Δ (delta). Through this transformation, SSMs possess a dual property:

• Recurrent mode during inference: The model processes a new token step by step through a constant hidden state, comparable to a traditional recurrent neural network (RNN). This results in O(1) time complexity per step and a constant memory footprint.
• Convolutional mode during training: Because the transformation matrices are linear and time-invariant, the entire sequence calculation can be written out as a global convolution. This makes full parallelisation on GPU clusters possible during the training phase.

Early implementations struggled, however, with the problem of gradient decay over long distances. The introduction of the structured state space sequence model (S4) by Albert Gu and colleagues solved this by structuring the transition matrix A using HiPPO matrices (high-order polynomial projection operators). HiPPO ensures that the hidden state retains a mathematically optimal reconstruction of the input history through orthogonal polynomials over a sliding time window.

The breakthrough of selective state systems (Mamba)

Although S4 proved that linear models can handle extremely long sequences, performance on complex natural language tasks lagged behind transformers. The fundamental cause lay in the strict time invariance: the matrices Ā, B̄ and C remained static and data-independent, regardless of the semantic content of the specific input tokens. A regular transformer can use attention matrices to focus dynamically on specific relevant tokens and ignore irrelevant stop words; a traditional LTI SSM treated every incoming token computationally with exactly the same filter.

With the introduction of Mamba (selective state spaces), this theoretical barrier was broken. Mamba makes the parameters B, C and the discretisation step Δ explicit functions of the current input vector x_t. This allows the network to modulate, per time step and depending on context, which information is absorbed into the compressed hidden state and which data is discarded immediately:

# Conceptuele selectiemechanica in Mamba:
B_t = Linear_B(x_t)
C_t = Linear_C(x_t)
Δ_t = Softplus(Linear_Δ(x_t) + Parameter_Δ)
A_bar_t = exp(Δ_t * A)
B_bar_t = (Δ_t * A)^(-1) * (A_bar_t - I) * (Δ_t * B_t)

# Recurrente toestandsovergang:
h_t = A_bar_t * h_(t-1) + B_bar_t * x_t
y_t = C_t * h_t

Through this dynamic selectivity, however, the model loses its strict convolutional form during training, because the convolution kernel is no longer time-invariant. To bridge this efficiency problem, the authors developed a hardware-aware parallel scan algorithm (associative prefix scan). Instead of writing intermediate states for each layer to the slower HBM memory (high bandwidth memory) of the GPU, the state transformations are calculated in fused form directly in the faster SRAM memory of the streaming multiprocessors.

Architecture comparison: transformer versus SSM

The structural differences between transformer attention mechanisms and selective state space models show up above all in operational costs, latency and scalability with documents of great length. The comparison below shows the algebraic and infrastructural properties:

When designing compact systems, clear points of overlap appear. The overview of the rise of small language models and local models makes clear that memory restrictions on edge devices weigh heavily. Because SSMs do not require gigabytes of KV cache for long documents, they run more efficiently on hardware with limited memory bandwidth.

Hardware impact and hardware-friendly implementations

The rise of alternative network architectures has direct consequences for chip design and compiler optimisations. Modern GPUs, such as the NVIDIA H100 and B200, are physically built around dense matrix multiplications (general matrix multiply, GEMM). The traditional self-attention mechanism maps algebraically onto these hardware blocks seamlessly. State space models, by contrast, lean more heavily by nature on sequential scans and element-wise operations, which historically were memory-bound rather than compute-bound.

In the broader context of the race for specialised AI chips and computing power we see chip suppliers and researchers moving towards each other. Mamba-2 recently introduced state space duality (SSD), a theoretical framework proving that selective state space transformations are equivalent to a specific form of semi-dense structured masking matrix multiplication. This allows Mamba-2 blocks to use the fast tensor core matrix units on GPUs directly, which raises training throughput by a factor of two to three compared with the first generation of Mamba.

Minimising data traffic between the slower central HBM memory and the fast on-chip SRAM memory of the processor remains the decisive factor. Where transformers have to load earlier keys and values from video memory continuously during every successive generation step, an SSM simply retains its compact internal state within the registers of the compute cores. This prevents inference servers from getting stuck on memory bandwidth at high concurrency.

Measurement methods and benchmark results

To quantify the performance of state space models objectively against transformers, the research community uses standardised evaluation protocols. Performance is measured along two axes: synthetic tasks for information recall and general language tasks.

Synthetic recall and the needle in a haystack test

A standard measurement method for long-context processing is the needle in a haystack (NIAH) benchmark. Here a specific, arbitrary fact (the "needle") is placed at varying depths (from 0% to 100% of the context length) within a substantial document of tens of thousands to millions of tokens (the "haystack"). The model is then asked to reproduce only that specific fact.

Transformers generally score 99% to 100% on this test, regardless of where the fact is hidden, thanks to direct pairwise attention. Pure SSMs, by contrast, show a characteristic degradation: facts sitting deep at the start of the context (early input steps) are sometimes overwritten in the hidden state when thousands of tokens of noise follow. At contexts above 32k tokens, the retrieval accuracy of a pure SSM without an attention mechanism drops significantly compared with models using self-attention.

Language understanding and perplexity

On academic language tasks such as MMLU (massive multitask language understanding), GSM8k (mathematical reasoning) and HumanEval (code generation), state space models perform at a comparable level to transformers of equal parameter size. When we measure perplexity (the degree of uncertainty in predicting the next token) across large text corpora, Mamba performs structurally better than earlier recurrent architectures such as LSTMs and approaches the scaling laws of strong transformer baselines.

The limits of state space models: where the model strains

Despite their theoretical superiority in computational complexity, state space models have not proved to be a universal replacement for transformers. In practical evaluations, pure SSMs show specific weaknesses inherently tied to their compressed state space:

• In-context retrieval and copying: Because an SSM has to compress all historical context into a fixed state size, the model struggles with exact repetition tasks (such as copying long code fragments verbatim) and needle-in-a-haystack searches in which one specific fact has to be dug out of an enormous text.
• State saturation: At extreme context lengths the fixed hidden state can become saturated. Information presented early in the context gradually fades in favour of more recent tokens.
• Loss of cross-document associations: Through self-attention, transformers can make direct links between arbitrary token pairs, regardless of the distance between them. SSMs have to channel these associations through sequential compression, which can lead to the loss of subtle logical correlations.
• Induction heads and copy mechanisms: Transformers naturally form so-called induction heads — specific attention mechanisms that recognise and copy patterns such as [A][B] ... [A] -> [B] directly. SSMs have to simulate this behaviour through continuous state transitions, which proves statistically less reliable on one-shot tasks.

Edge cases and failure modes in production

In business and technical production environments, specific edge cases arise in which developers have to take account of the properties of state space models. The main failure patterns include:

Repetitive code and data structures

When a model has to analyse a huge JSON file, CSV dump or log file in which thousands of lines resemble one another structurally but contain subtle numerical deviations, the selective state of an SSM can conflate them. Because the selection parameters Δ and B are steered by token vectors with high semantic similarity, the state becomes oversaturated with redundant structural patterns, so that rare deviations are lost.

Long-range reasoning with intermediate steps

In complex logical deduction tasks where an assumption on page 2 has to be linked to a conclusion on page 80, the quadratic attention of a transformer functions as a direct lookup register. An SSM has to carry the premises continuously through dozens of state transformations. Without periodic attention mechanisms, the signal-to-noise ratio of that early premise can erode before the conclusion is reached.

Cost-benefit analysis for large-scale deployment

The operational trade-off between transformers and state space models translates into a clear cost structure for cloud infrastructure and inference servers. The saving from SSMs shows up mainly at three levels:

• Inference throughput (tokens per second): Because SSMs do not have to exchange a KV cache between HBM and compute cores, generation speed with long contexts rises by a factor of three to five. A single GPU can serve considerably more simultaneous users without video memory filling up.
• Hardware investments (VRAM footprint): For a context window of 128,000 tokens, a classic 70B-parameter transformer requires tens of gigabytes of dedicated VRAM purely for the KV cache of a single active session. A comparable SSM model gets by with a fixed state of only a few megabytes per stream, which drastically lowers server costs.
• Latency stability: With transformers the time per generated token rises linearly as the answer grows longer (because of the expanding attention matrix). With state space models the generation speed stays exactly the same from the first to the hundred-thousandth token (O(1) per step).

Hybrid architectures: the best of both worlds

Because of the specific limitations of pure SSMs in information retrieval and their unmatched computational advantages in sequence processing, the research climate has moved largely towards hybrid models. Models such as Jamba (AI21 Labs), RecurrentGemma (Google) and various open-weights initiatives combine transformer attention layers, Mamba blocks and mixture of experts (MoE) in one composite topology.

In such a hybrid architecture the network handles, for example, 80% to 90% of the layers through selective state spaces, while every fourth or eighth layer contains a traditional self-attention layer. This brings considerable advantages:

• The total size of the KV cache falls by 70% to 85%, making enormous batch sizes and contexts of hundreds of thousands of tokens possible on standard hardware infrastructure.
• The periodic attention layers retain the superior in-context retrieval and reasoning quality of transformers, so that needle in a haystack scores remain virtually perfect.
• Overall inference latency drops drastically because the majority of token generations proceed linearly and without heavy memory I/O.

Outlook and synthesis

State space models mark a fundamental shift in how the AI industry views sequential processing. Where for almost a decade the transformer model counted as the only valid blueprint for neural networks, Mamba and state space duality prove that linear complexity is viable without heavy quality loss on complex language tasks.

Although pure SSMs do not yet fully replace the transformer mechanism on specific precision and lookup tasks, hybrid architectures are proof that the hegemony of pure self-attention has been definitively broken. For real-time processing of audio signals, analysis of millions of lines of source code, sensor data in industrial environments and locally running assistants with low latency, state space models offer a mathematically elegant and economically necessary foundation for the coming generations of language and sequence models.