Subquadratic LLMs: cheaper long context for on-prem RAG
Subquadratic models lower the cost of long context. A real lever for on-prem RAG, but still young.
This translation was produced automatically using AI. The German version is the editorially reviewed original.
Subquadratic models are a real lever for the economics of long context on your own hardware. Bounded memory for the key-value cache and two to three times the throughput on long inputs are physics, not marketing. But in 2026, the value for Mittelstand RAG still comes predominantly from retrieval quality and a solid, well-supported model, not from betting on a young architecture.
This piece explains why long context is expensive, which subquadratic approaches exist, what the loudest promises actually have evidence for, and what that concretely brings for on-prem RAG. The recommendation at the end is unspectacular, and viable for exactly that reason.
01. Why long context is expensive
The reason lies in the mathematics of self-attention: its cost grows quadratically with the length of the input.
In a classic transformer, every token is compared against every other token. If the context length doubles, the compute cost for attention roughly quadruples. On top of that comes memory: the key-value cache that the model carries along during generation grows linearly with length and quickly eats up the available memory on a fixed GPU. That is the real bottleneck for on-prem operation, where the hardware is fixed and cannot be elastically rented on top.
This is exactly where subquadratic approaches come in. They try to push sequence cost towards linear, so O(n) instead of O(n squared), and to cap the memory per token. What this means for the hardware reality inside a company, we worked through in local LLM.
02. The subquadratic landscape
There are several paths, and the practical winner of the years 2024 to 2026 is not the purest one, but the hybrid one.
- State space models (Mamba). Mamba (December 2023) and Mamba-2 (May 2024) model sequences in linear time with a fixed state memory instead of a growing cache (arXiv). The price: a fixed state is a lossy compressor and weaker at exactly recalling positions far back in the sequence.
- Linear attention (RWKV, RetNet). Reformulate attention so that cost grows roughly linearly. Similar trade-off on associative recall.
- Hybrids in production. No one ships pure subquadratic models at top quality. Instead, everyone mixes a minority of full attention layers with a majority of state space or linear layers: Jamba (AI21, 2024), MiniMax-01 (January 2025), Falcon-H1 (May 2025), NVIDIA's Nemotron-H, IBM Granite 4.0 (October 2025) and Qwen3-Next (September 2025). The full layers preserve memory, the lean ones buy the cost and memory advantage.
The NVIDIA line behind Nemotron, which uses exactly this hybrid design, we described in Nemotron 3.
03. What 'subQ' promises, and what of it is proven
The term calls for caution, because it carries two meanings, one architectural and one marketed.
Alongside the general subquadratic family there is a concrete product: the start-up Subquadratic came out of stealth in May 2026 with funding of around 29 million dollars and promotes a subquadratic, sparse attention mechanism. The headlines are big: a research model with 12 million tokens of context, around a fifth of the cost compared to top models, a multiple of the speed (The New Stack).
The honest assessment: part of it is verified by third parties, for instance a RULER-128K result, while the spectacular figures on 12 million tokens and the speed factor are vendor claims from single runs and not independently reproduced. That is a promising signal, not a basis for procurement. This exact distinction, proven versus claimed, is the discipline we apply to every new model.
04. What this brings for on-prem RAG
The concrete gain is memory and throughput on fixed hardware, and that is exactly the right lever for on-prem RAG.
- Bounded cache, predictable memory. The state space layers carry a fixed state, so memory does not explode with length. IBM states over 70 percent less requirement for long context with Granite 4.0, a vendor figure, but plausible, and pointing towards more concurrent sessions per GPU.
- Model and context on one card. NVIDIA reports that a Nemotron-H model with 51 billion parameters, cache included, fits onto a single 80 GB card, again a vendor figure.
- Looser chunking discipline. Cheaper context allows larger windows and less anxious slicing of documents.
Honest caveats belong here too. For memory-heavy RAG, for instance the exact passage across very long texts, pure subquadratic models still lag behind full attention, hybrids close the gap largely but not completely. The tooling around them, meaning quantisation, serving and fine-tuning, lags behind the established dense models. And the most important sentence: long context is not the same as good retrieval. A solid model with good RAG today beats, almost always, the attempt to simply dump in millions of tokens. What RAG actually delivers is covered in What is RAG and The power of RAG.
05. How I treat this today
Subquadratic is worth watching, but not worth betting on. Here is how I handle it:
- Build the RAG layer model-independent. Behind a stable interface, so that a Granite-4-class hybrid can be dropped in as soon as the tooling around it is mature.
- Start with a solid, well-supported model, not the newest architecture, and get the value from retrieval quality.
- Ignore unproven promises. A claimed factor of a thousand can wait until someone independent reproduces it.
That is the same pattern we apply to on-prem versus cloud: decide deliberately, build swappable (on-prem vs. cloud LLM). The interesting question is not whether subquadratic models will one day make long context cheap. They probably will. The question is whether your architecture is ready to capture the gain without betting on it today. Is your RAG layer built so that you can swap the model without rewriting everything?
