Qwen 3.8 Flash Next: 125B Qwen4 Preview, 6B Active

Qwen4 architecture preview with sparse attention: 125B, 6B active

Active parameters are the share of a model's weights that take part in handling each token. Qwen3.8-Flash-Next holds 125B parameters and activates 6B of them per token.

Two further blocks sit outside that count. One is 51B of n-gram embedding, a lookup table keyed by short runs of tokens. The other is 4B of MTP, short for multi-token prediction.

Configuration from the model card

Item	Value
Total parameters	125B, 6B activated
Additional parameters	51B n-gram embedding, 4B MTP
Hidden dimension	2560
Layers	48
N-gram embedding	20,000,000 entries, bigrams and trigrams at layer 2
Experts	512, with 10 routed plus 1 shared active

It is a 48-layer model with 512 experts, and each token goes to 10 routed experts plus 1 shared expert. The card lists the type as a causal language model with a vision encoder.

Sparse attention at micro-block level in the Qwen4 architecture

Attention is the step where a model looks back over the text so far. Sparse attention looks at only part of it. Qwen's card names four changes that make up the new design:

• Hybrid attention. Gated DeltaNet is now paired with Qwen Sparse Attention (QSA), replacing the earlier pairing of Gated DeltaNet and Gated Attention.
• Gated Residual. A read gate and a write gate control the information flowing through widened residual streams.
• N-gram Embedding. Parameters scale through lookups of short token sequences.
• Training recipe. The Muon and AdamW optimizers are applied to different categories of weights.

Of the four, QSA works on micro-blocks rather than on single tokens. Qwen says this cuts long-context latency significantly and calls that a critical gain, since agentic workloads (a model working through many steps on its own) increasingly dominate real-world usage. The latency claim is Qwen's own.

Qwen4 architecture context: 262,144 tokens with sparse attention

A token is the word fragment a model reads as one unit. Qwen3.8-Flash-Next takes 262,144 tokens natively, and the card says that can be extended to 1,000,000 tokens. Going past the native length takes a configuration change, and the card advises modifying the rope_parameters setting only when long contexts are actually needed.

Qwen separates the open weights from the product. The hosted Qwen3.8-Flash is the official version built on Qwen3.8-Flash-Next, with the 1M window on by default and official built-in tools. If you want the long window without touching the configuration, start from that version.

Sparse attention benchmarks for the Qwen4 architecture preview

The card publishes a comparison table against other models, and it publishes the conditions behind the numbers. Those conditions decide how far the scores can be trusted.

Published scores for Qwen3.8-Flash-Next

Evaluation	Score
DeepSWE 1.1	58.7
SWE-bench Pro	62.5
SWE-bench Multilingual	81.0
CoWorkBench	73.9
IFBench	81.3
GPQA Diamond	91.7
LiveCodeBench v6	91.9

The card attaches conditions to four evaluations. Each footnote limits what the number can be compared with:

• DeepSWE 1.1 ran through two harnesses, Claude Code and mini-SWE-agent, and the "Published scores for Qwen3.8-Flash-Next" table reports the higher score of the two. The card adds that Qwen3.8-Flash-Next did best on mini-SWE-agent.
• SWE-bench Pro ran every model on the Claude Code harness at temp 1.0, top_p 0.95 and a 256K window. The exception is Claude-Opus-4.6 (Max), whose officially published score is used. Problematic tasks were corrected, and all baseline models were re-evaluated on that refined version.
• HLE, which has no row in the card's comparison table, was judged by GPT-4o.
• CoWorkBench is an in-house benchmark for long-horizon office and productivity agent tasks across computer science, finance, law, medicine and other domains.

None of this makes the scores wrong. It does mean they are Qwen's results under Qwen's settings. A figure from another lab that used a different harness or grader would rest on different conditions, so read these scores as claims about this model, not as a ranking against others.

Open weights: Qwen4 architecture and sparse attention preview

Qwen released the weights and configuration files in the Hugging Face Transformers format. The card shows the license as qwen-community-1.0 and lists Transformers, vLLM, SGLang and TokenSpeed as compatible frameworks.

The card frames the release as a preview, not a finished model. It describes an experimental preview of the architecture that will underpin Qwen4, and it welcomes feedback as Qwen builds what comes next.

For the rest of the series, see Qwen3.8 Open Weights: What You Actually Get. Memory needs for Qwen3.8-27B are covered in Qwen3.8-27B VRAM: 7GB to 56GB by Quant Level, and the choice of quantized files is covered in Unsloth GGUF Dynamic 3.0: Where Small Quants Break.