Unsloth GGUF Dynamic 3.0: Where Small Quants Break

What Unsloth Dynamic 3.0 GGUFs changed from 2.0

Quantization stores a model's weights at lower precision, so the file gets smaller and needs less memory to run. GGUF is the single-file model format that llama.cpp and other local inference engines load, so an Unsloth GGUF is a model that Unsloth has quantized and packaged for running locally.

Dynamic 3.0 is a new quantization recipe that changes how the files are quantized, not how the model is trained. Unsloth's documentation calls it the next iteration of Dynamic and a major improvement over Dynamic v2.0, and says the files work with most inference engines, llama.cpp and Unsloth Desktop included.

The release updates an early preview Unsloth had shared before. The docs page gives no release date for it.

The headline claim concerns top-1 accuracy, meaning how often the quantized model's single most likely next token matches the original's. Unsloth says its Qwen3.8-27B quants beat every other provider by more than 10% on that measure at the same size.

New imatrix calibration data and layer selection

Three things changed: the imatrix calibration dataset, the layer selection, and the number of quantization techniques. An imatrix is a file of statistics gathered by running sample text through the model, and the quantizer uses it to decide which weights deserve more precision.

Unsloth now builds that sample text from diverse sources and says it is refined for agentic coding, chat and multilingual performance. It also improved layer selection, which decides how much precision each layer keeps. And it added many more quantization techniques, without naming them on the page.

The imatrix file itself is public. It is imatrix_unsloth.gguf in the Qwen3.8-27B-GGUF repository on Hugging Face, about 13.6MB.

That is one file of statistics rather than the calibration text or a record of what each quant was tuned on. Unsloth encourages researchers to build variations and fine-tunes of Qwen3.8 from its quants and imatrix.

Why the new quants involve no retraining

The new quants involve no retraining: Unsloth states that everything is post-training quantization, or PTQ. PTQ means the finished model's weights are compressed after training ends. Unsloth says it does not train on the imatrix calibration dataset and uses neither QAT (quantization-aware training) nor QAD, the two routes that keep training the model while it is being quantized.

The page says this about Unsloth's own quantization pipeline and nothing more. It does not say the recipe carries over to other models, so do not read the absence of retraining as a promise about anything beyond the Qwen3.8-27B files.

Where the older Dynamic 2.0 recipe still applies

Larger quants still use the old Dynamic 2.0 recipe, because the new method helped the smaller files far more than the bigger ones. Unsloth compared UD-3 with its older UD-2 on unseen Wikitext and code and saw a large drop in KL divergence (KLD), the gap between a quant's outputs and the original's, on the smaller quants, and much less on the bigger ones. It says it plans to experiment on the larger quants too.

The page does not say where "smaller" ends and "larger" begins. For background, Dynamic 2.0 adjusted the quantization type of every possible layer, used a hand-curated calibration set of more than 1.5M tokens, depending on the model, and gave each model its own scheme.

GGUF quantization comparison: file size by quant

The quantized Qwen3.8-27B GGUFs span 6.19GB to 31.46GB, and the two per-file top-1 figures Unsloth gives both sit at the small end. File sizes are from the Hugging Face repository, and accuracy figures are from Unsloth's documentation.

File sizes from UD-IQ1_S to UD-Q8_K_XL

On Hugging Face the quantized files run from 6.19GB for UD-IQ1_S to 31.46GB for UD-Q8_K_XL. The sizes come from the unsloth/Qwen3.8-27B-GGUF file listing as of September 30, 2026, in decimal GB. The unquantized BF16 copy sits in a separate folder and is not in the chart.

Qwen3.8-27B GGUF file size by quant (GB)

Read from the Hugging Face file listing on September 30, 2026. A selection: the repository holds further quants, such as UD-Q5_K_XL and UD-Q6_K_XL. Q4_0 and Q8_0 carry no UD- prefix; UD-Q2_K_XL is highlighted.

6.19UD-IQ1_S

6.73UD-IQ1_M

7.27UD-IQ2_XXS

8.37UD-IQ2_S

9.83UD-Q2_K_XL

10.93UD-IQ3_XXS

12.04UD-IQ3_S

13.15UD-Q3_K_XL

16.06Q4_0

17.56UD-Q4_K_XL

29.05Q8_0

31.46UD-Q8_K_XL

0102030

Unsloth's page attaches top-1 figures to only two of these files, UD-IQ1_S and UD-Q2_K_XL, and it says the four files up to UD-IQ2_S have had the MTP module removed. For the other files the page gives no top-1 figure, so the chart compares size alone.

What UD-Q2_K_XL buys at 9.83GB

Unsloth reports UD-Q2_K_XL, at 9.83GB, as about 8% more accurate on top-1 than the next best. The sentence does not say which file or provider the next best is.

Unsloth adds that this quant produced a working HTML program with one small JavaScript bug where, Unsloth says, the result would previously break, and the page shows an illustration generated with a 2-bit Qwen3.8-27B GGUF. That is one task, useful as a signal and not a benchmark. The smaller UD-IQ1_S, at 6.2GB without the MTP module, keeps around 72% top-1 accuracy and is 89% smaller than the full model, by Unsloth's numbers.

Why small quants drop the MTP module

Unsloth removed the MTP module from every quant at UD-IQ2_S and below, which saves around 500MB. MTP is multi-token prediction; the Qwen3.8-27B model card lists it as an architecture feature trained with multiple steps. If you need it, Unsloth says a separate Q4_0 MTP module is available.

The comparison plots also take the MTP module, which the page calls the MTP head, out of the size axis for every provider, to keep the comparison fair. For files that still include the module, plotted size and downloaded size can therefore differ.

Why top-1 accuracy alone can mislead

Top-1 accuracy scores one prediction at a time, and Unsloth itself says that is not really effective for gauging actual inference. A quant can match the original's next token and still wander off two tokens later, and a single-token score cannot see that.

What Divergence-300 @32 measures

Divergence-300 @32 compares every quant with the BF16 original over 32 generated tokens instead of one. BF16 is the unquantized, full-precision model. At each step both models pick their single most likely token, which is greedy argmax decoding, and the test asks whether the quant stays on the original's path.

Unsloth calls this a better metric than top-1 accuracy because it stretches KLD top-1, which looks at one token, to something closer to KLD top-1 over 32 tokens. KL divergence measures how far the quant's token probabilities sit from the original's, and 0 would mean identical. Unsloth ran KLD benchmarks for all providers and reports top-1 and mean KLD.

Held-out prompts and 32-token trajectories

The 300 prompts are held out, meaning none of them is in the calibration dataset. They come from Terminal-Bench 2.1, DeepSWE, Harbor, MathArena 2025-26, and non-Latin and long-document prompts. Unsloth ran BF16 against all quants and all providers on those same 300 prompts.

Thirty-two tokens is a short window. It catches early drift but says nothing about a long agent run.

Do Unsloth quants overfit their calibration data?

Unsloth says its 3.0 quants do not overfit, but the evidence is Unsloth's own testing. A quant overfits when it is tuned to score well on data that resembles its calibration set. Unsloth's own v2.0 notes describe the trap: calibrate on Wikipedia-related text and test on Wikipedia data, and the quant looks better than it will elsewhere.

The overfitting controls Unsloth describes

Unsloth describes four controls against overfitting. Together they aim at one worry: a quant that scores well only because the test looks like its calibration data.

• Calibration and testing use totally different datasets, with leakage removed as much as possible.
• KLD is tested on unseen datasets, Wikitext and code.
• There is no QAT or QAD, only PTQ, which Unsloth says makes overfitting less of a concern than in those approaches.
• Divergence-300 @32 uses 300 unseen prompts as a further check, and Unsloth says it shows the new UD-3 methods do not overfit.

What Unsloth's own numbers cannot show

Every accuracy figure cited is Unsloth's own measurement, not a third-party re-test. Treat each one as the vendor's number until someone else reproduces it.

The headline gain is also worded two ways on the same page. The release announcement says more than 10%, while the benchmark section gives an upper limit instead: at most 10% more top-1 accuracy for the same disk space, most visible on the smaller quants. The page does not reconcile the two, so treat the gain as what Unsloth claims and not as a result to expect on your own files.

The wording differs between sources: the docs page says "top-1" while the repository README says "top-1%". The docs page wording is used throughout.

Where 1-bit quantization breaks LLM agents

Below UD-Q2_K_XL, Unsloth's own 32-token benchmark shows a sharp accuracy drop, and Unsloth says tool calling and non-thinking mode break down there. The page section is titled "1-bit should not be used for agentic use-cases", but the drop it measures sits between UD-Q2_K_XL and UD-IQ2_S.

UD-IQ2_S is an IQ2 quant, and the files Unsloth calls UD-1bit are UD-IQ1_S and, in the file list, UD-IQ1_M. UD-IQ2_S is therefore grouped as "below UD-Q2_K_XL", not as 1-bit.

The drop from UD-Q2_K_XL to UD-IQ2_S

On Divergence-300 @32, accuracy falls from around 25% at UD-Q2_K_XL to under 8-10% at UD-IQ2_S. Both figures are for 32-token prediction, and Unsloth reads the gap as the point where tool calling and non-thinking modes stop working.

Divergence-300 @32 accuracy for 32-token prediction, as Unsloth reports it (%)

Unsloth gives UD-IQ2_S only as "under 8-10%", so its bar is drawn at 10, the top of that range.

01020

~25UD-Q2_K_XL

under 8-10UD-IQ2_S

Non-thinking mode is when the model answers without a reasoning pass first. The Qwen3.8-27B model card says thinking is on by default and can be switched off per request.

Tool calling is the model emitting a structured request for an outside tool to run. Unsloth says both stop working at that drop. An agent loop depends on tool calling, so that is the part that matters for agents.

Looping, empty replies and failed tool calls

Unsloth lists three failure modes on the smallest quants, and only the first two come with a workaround. The table lays out each symptom, where it shows up and what Unsloth advises.

Symptoms of the smallest quants and Unsloth's advice

Symptom	Where it appears	Unsloth's advice
Looping	Files smaller than UD-Q2_K_XL	Set presence_penalty to 1.5 or more
Empty responses	1-bit files	Keep thinking on, even at low reasoning effort
Failed tool calls	Heavily quantized files	Keep the model away from tool calling

Presence penalty is a sampling setting that discourages the model from repeating tokens it has already used. The Qwen3.8-27B model card, repeated in the Hugging Face README, cautions that a higher value may occasionally mix languages and slightly lower performance, so raise it only as far as the looping requires.

For tool calls the page gives a plain warning instead of a fix. Only general knowledge is retained when the model is quantized heavily, and it will either fail to call tools, keep calling tools or not call them at all.

What the smallest quants can still answer

Very short general-knowledge fact questions still work on the smallest quants, and even then Unsloth says UD-Q2_K_XL is the better pick. Its wording is narrow: very short fact questions, and nothing about tools or long outputs.

Unsloth also says a top-1 recovery of 77% is no substitute for a Divergence-300 @32 score of 8%, which it calls the better indicator for actual inference workloads. The page does not say which quant those two figures belong to. Even without a file name, the pair shows that a good-looking top-1 number and a poor 32-token number can describe the same kind of file.

Which Unsloth quant should you run?

If the job calls tools, run UD-Q2_K_XL or larger, because that is Unsloth's own cutoff. Which Unsloth quant fits you depends on three things: whether tools are involved, how much memory you have, and whether the numbers hold on your own work.

Pick by whether the job calls tools

For anything that calls tools, Unsloth's cutoff is UD-Q2_K_XL at 9.83GB. Below it, the docs page reports looping and the sharp accuracy drop, and it tells readers not to use the heavily quantized model for tool calling. Unsloth does not name a minimum quant for agents in those words; the cutoff is where its warnings begin.

Pick by the memory you have

If memory is the limit, choose from the size chart above, and treat anything smaller than UD-Q2_K_XL as a fit for short factual questions only. File size is not the whole memory bill. Qwen3.8-27B VRAM: 7GB to 56GB by Quant Level covers what each quantization level needs in total and what a long context adds on top.

Check the numbers on your own workload

Because every accuracy figure comes from Unsloth's own testing, run the quant on your own prompts before you commit to it. Send the same tool-calling task to UD-Q2_K_XL and a larger file, and watch for looping, empty replies and calls that never happen. The published imatrix file lets you inspect what Unsloth used, though it is a statistics file and not the calibration text.

If you are weighing a training-based route instead, LFM2.5 DSpark and QAD: Faster Decode, Better 4-bit covers QAD, the approach Unsloth says it does not use. For an Unsloth GGUF, the practical rule is short: stay at UD-Q2_K_XL or above for tools, and test the rest yourself.