Attention Is Not What You Need? Hugging Face downloads say 0.4% agree

Short answer: The paper "Attention Is Not What You Need" hit the Hacker News front page this morning, but the open-model market already voted. We pulled 30 days of Hugging Face downloads by architecture: models with no attention at all got 1.04 million downloads, about 1 in every 263. Attention is not dying. It is being diluted, and NVIDIA is the one doing it.

TL;DR: Across 20 architecture tags and about 12,000 models, plain Transformers pulled at least 254.0 million downloads in the last 30 days, hybrids pulled 16.9 million and pure attention-free models pulled 1.04 million. A single NVIDIA hybrid, Nemotron 3 Nano 4B, got 1.67 million downloads on its own, more than every Mamba, RWKV and xLSTM model on the Hub combined. The real fight is not attention versus no attention; it is how little attention you can get away with, and the hybrids are winning it 16 to 1.

Key numbers

What	Number	Source, checked Oct 26 2026
Transformer downloads, top 1,000 models in each of 5 families (floor)	254.0M	Hugging Face Hub API
Hybrid (Mamba or linear layers plus some attention) downloads	16.9M	Hugging Face Hub API
Pure attention-free (Mamba, Mamba2, RWKV, xLSTM) downloads	1.04M	Hugging Face Hub API
Share of all counted downloads with no attention	0.38% (1 in 263)	Tensorust analysis of the pull above
NVIDIA Nemotron-H share of hybrid downloads	65% (11.0M)	Hugging Face, NVIDIA
Grassmann model perplexity gap vs same-size Transformer	10 to 15% worse, at 13 to 18M parameters	arXiv 2512.19428

The paper says attention is optional. Its own numbers say not yet

The paper, by Chong and Zhang, swaps the attention matrix for what it calls a Causal Grassmann layer. In plain terms: instead of every word comparing itself to every other word (the L by L attention grid that makes long context expensive), it squeezes nearby word pairs into small geometric shapes and mixes those back in.

The pitch is linear cost in sequence length and a model you can actually describe with math. The results are honest and small: on Wikitext-2, the Grassmann models with 13 to 18 million parameters land within about 10 to 15% of a same-size Transformer's perplexity (lower perplexity means better next-word guessing).

That is a toy scale. Llama 3.1 8B is more than 400 times bigger than the largest model in the paper. Being 10% worse at 18 million parameters is a research result, not a replacement, and HN's 131 points this morning are mostly people who have heard "attention is dead" before.

The market already ran this experiment, and here is the bill

Mamba and RWKV are not new; their oldest checkpoints on the Hub have been downloadable for years. Both promised the same thing as this paper: no attention, linear cost, long context for cheap. So we asked the only judge that pays its own way: who downloads them.

We queried the Hugging Face Hub API on Oct 26 2026 for 20 architecture tags, sorted by downloads, and de-duplicated models across tags. Downloads on the Hub are a rolling 30-day count. The full list is in the data download at the bottom of this page.

All the pure attention-free models together got 1.04 million downloads. The top one, state-spaces/mamba-130m-hf, is a 130 million parameter checkpoint people use for tests and tutorials. xLSTM 7B from NX-AI got 4,696.

For scale: Qwen3-0.6B, one small Transformer, got 23.7 million downloads in the same window. That is 23 times the entire attention-free field.

The gap is not about quality claims; it is about everything around the model. Fine-tuning scripts, quantized builds, inference servers and tutorials all assume attention. A model without it starts every week behind on tooling, and downloads follow tooling. That is the moat the Grassmann paper would also have to cross, at a size more than 400 times bigger than it tested.

Hybrids are where the money is, and NVIDIA owns them

The interesting number is in the middle. Hybrids keep a few attention layers and replace the rest with Mamba or linear layers that cost the same no matter how long the input is. They pulled 16.9 million downloads, 16 times the pure attention-free models.

Follow the money and it leads to one company. NVIDIA's Nemotron-H family is 11.0 million of those 16.9 million, about 65%. Its top checkpoint, Nemotron 3 Nano 4B, did 1.67 million downloads by itself. Alibaba's Qwen3-Next (2.53 million) and Liquid AI's LFM2 (2.52 million) are the only other hybrids above a million.

Our read: NVIDIA does not care which architecture wins, as long as the winner runs best on its chips and ships with its name on the card. Giving away a fast hybrid is how you make sure the post-attention era, if it ever comes, is still an NVIDIA era. Nobody has proven that is the plan. The download chart is consistent with it.

Who wins, who loses

• Wins: NVIDIA. It owns two thirds of the only non-Transformer category with real traction, for free.
• Wins: anyone running long context on their own hardware. Hybrids keep memory flat as prompts grow, which is exactly where the cloud API bill hurts.
• Loses: pure attention-free startups. RWKV, the longest-running open alternative, got 226,000 downloads across its main tag. That is about 1% of one Qwen checkpoint.
• Loses: the "attention is dead" headline. Every year it gets written. Every year the share stays under 1%.

How we counted, and where it undercounts

We pulled the top 1,000 models by downloads for each tag. For the five big Transformer families (Llama, Qwen2, Qwen3, Gemma 3, Mistral) that cap cuts off the long tail, so 254.0 million is a floor and the real attention-free share is even smaller than 0.38%. We did not count GPT-style families we did not query, which also pushes the share down.

Tags are set by model authors, so some are noisy: a few fine-tunes of other models carry the mamba tag. Downloads count every pull, including CI test fetches, which inflates tiny test checkpoints on every side. Treat the ratios, not single numbers, as the signal.

What this means for you

If you build on open models, do not bet a product on a pure attention-free model this year; the tooling, fine-tunes and support all live on the other 99.6%. If long prompts are what make your bill hurt, test a hybrid like Nemotron 3 Nano or Qwen3-Next against what you run now, because that is the one place the "no attention" idea is already paying off. More on picking models in our LLM hub, the LLM price war breakdown, our Kimi K3.1 open-weights piece and every story on the news page.

Want the number for your own stack first? Our AI Cost Audit prices your real AI API bill on 71 models and shows what a switch saves. We keep fixing it at no extra cost until it does what it promises.

Sources

• Chong and Zhang, "Attention Is Not What You Need", arXiv 2512.19428, read Oct 26 2026
• Hugging Face Hub models API, 30-day downloads for 20 architecture tags, pulled Oct 26 2026
• NVIDIA, Nemotron 3 Nano 4B model card, Oct 26 2026
• Qwen, Qwen3-0.6B model card, Oct 26 2026
• Hacker News front page, "Attention Is Not What You Need (2025)", 131 points, Oct 26 2026