DiffusionGemmaとは？拡散言語モデルと自己回帰モデルの違い

"We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed."（Abstract）／"Autoregressive (AR) models dominate the current Large Language Model (LLM) landscape"（1 Introduction）— arXiv より

"Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models."（Abstract）／"Canvas generation starts from a canvas of uniformly random tokens and is iteratively denoised"（3.1）／"Because all tokens can transition between one another, the model can continuously correct its own errors: tokens accepted during earlier denoising steps"（3.2 Multinomial diffusion。2つ目の断片のあいだに数式記号が入る）— arXiv より

"When serving many requests simultaneously, one can achieve acceptable throughput by batching"／"serving single or low-concurrency requests is fundamentally memory-bound: time spent transferring model weights and context KV cache from memory to the accelerator far exceeds time spent on actual computation."（1 Introduction 冒頭。2つのあいだに別の語句が入る）— arXiv より

"Text diffusion circumvents this bottleneck by predicting entire blocks of tokens simultaneously"／"effectively shifting execution from a memory-bound regime toward a compute-bound one."／"DiffusionGemma generates on average 20 TPF"（以上 1 Introduction。断片のあいだに図番号や別の語句が入る）／"Because this denoising process is factorized, it assumes conditional independence during individual reverse steps."／"This structural tradeoff can occasionally introduce local inconsistencies or conflicting grammatical predictions."（以上 2）／"The model denoises a canvas of 256 tokens at a time, and once a canvas is fully denoised, it is committed to the sequence history, and the model begins denoising the next canvas."（3.1）— arXiv より

"Some models, e.g., Gemini Diffusion and Mercury, are locked behind proprietary APIs. On the other hand, existing open-weights alternatives either exhibit limited reasoning capabilities and multimodal understanding, or fail to deliver on the extreme latency benefits promised by the diffusion technology, or both. Until now, there has been no text diffusion model that is highly intelligent, exceptionally fast, and openly accessible."（1 Introduction）— arXiv より

"Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters."／"Our compute-efficient two-stage training pipeline uses fewer than 10%"（以上 Abstract。2つ目の断片の直後に続きの語句が入る）／"We warm-start DiffusionGemma from the final post-trained, publicly released weights of the Gemma 4 26B A4B MoE model"（1 Introduction）／"Only a moderate amount of SFT is needed to achieve good performance in non-thinking mode, however extended SFT is crucial for the model to learn thinking behaviour."（4 の図6の説明文）— arXiv より

"it yields a 10-point improvement on the combined GPQA-Diamond and LiveCodeBench-v6 score. On the efficiency axis, it quadruples the TPF from 5 to nearly 20, unlocking ultra-low latency inference."／"This counterintuitive behavior stems from the SFT model frequently degenerating into repetitive token loops in this restricted-step regime."／"it encourages the model to produce concise, token-efficient outputs"（5。断片のあいだに別の語句が入る）— arXiv より

"By preserving features of its AR starting point, DiffusionGemma exhibits strong multimodal understanding, long-context capabilities, and thinking mode, enabling it to generate reasoning traces prior to responding."（1 Introduction）／"Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding."（Abstract 末尾）— arXiv より

"it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding."（Abstract）／"Assuming a TPF of 19.74"／"the average decoding throughput of the model is 1456 TPS"（以上 6 Throughput for batch size of 1 serving。断片のあいだに別の語句が入る）／"early benchmarking by third-party inference providers such as Unsloth"／"demonstrates these speeds can scale up to 2,000 TPS on the NVIDIA RTX 6000."（以上 1 Introduction。断片のあいだに年の表記が入る）— arXiv より

"Speculative decoding can improve utilization by drafting a candidate sequence, typically 8 tokens, from a small “drafter” model and feeding the draft to the LLM for verification"／"With a draft length of 8 this can produce 3-6 tokens per forward pass (TPF)"／"However, the draft-then-verify paradigm is still limited: on one hand, AR drafters are bottlenecked by sequential generation"（以上 1 Introduction。断片のあいだに参照番号が入る）／"By successfully bypassing redundant refinement steps, this mechanism allows the model to dynamically scale its inference-time compute to the complexity of the prompt."（3.3）— arXiv より

"DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability."（Abstract）／"it extends the speed frontier beyond both prior text diffusion models and the Gemma 4 AR family across all parameter scales (from E2B up to 31B), even when the AR models are equipped with multi-token prediction"（1 Introduction）／"Quality and output speed are calculated as the average across GPQA-Diamond and LiveCodeBench-v6 for all models."（Figure 1 の説明文）— arXiv より

"Although adapting the model to the diffusion regime introduces a performance penalty compared to the original AR baseline, the massive gains in inference speed offer an entirely new operating point for latency-critical applications."（1 Introduction）／"it encourages the model to produce concise, token-efficient outputs"（5）— arXiv より

"Tying our architecture to an existing AR backbone is a pragmatic choice that trades off absolute generation quality for a substantial reduction in the computational cost of training."（3）／"However, we note that tokens from previously generated canvases are permanently frozen."（3.2 Multinomial diffusion）／"its performance scores in AR mode land squarely between those of"（3.5。この直後に続きの語句が入る）— arXiv より

"In the low batch size regime, DiffusionGemma offers substantially higher TPS per user and higher total throughput."／"It is only at moderate batch sizes (around 32 concurrent requests) that AR models begin to have a throughput advantage."（以上 Figure 12 の説明文）／"these results are without targeted optimization for batch sizes larger than 1: current kernel selection is suboptimal, and sampling has not been tuned to scale with batch size"（6 Multi-user throughput）— arXiv より

"We release DiffusionGemma with open weights under a permissive Apache 2.0 license, aiming to democratize access to state-of-the-art text diffusion technology."／"the liberal licensing removes friction, ensuring seamless integration into both experimental prototypes and commercial applications without restrictive usage barriers."／"Despite the model being available for only a few weeks at the time of writing, it is already serving as the engine for specialized downstream applications."（以上 1 Introduction）／"Reference inference implementations are available in HuggingFace Transformers"（6 Low batch size serving）— arXiv より