Unslothが公開する量子化モデルの精度と、1ビットまで削った時の限界

"Unsloth Dynamic v3.0 is the next iteration of our Dynamic quantization and a major improvement over Dynamic v2.0."／"This is an update of our first shared early preview version of Dynamic v3.0."／"We now use a much higher-quality imatrix calibration dataset from diverse sources. The dataset is refined for agentic coding, chat, and multilingual performance. We also improved layer selection and introduced many more quantization techniques to preserve as much model quality as possible."／"We do not train on the imatrix calibration dataset, and we do NOT use QAT or QAD. Everything is done through post-training quantization."／"When comparing to our older UD-2 on unseen Wikitext and Code, we show great improvement on KLD - the bigger ones not so much, so we still use our old UD-2 for the larger quants - we plan to experiment and improve them as well!"（冒頭 ／ 手法の節 ／ Not Overfitting 節）— Unsloth Documentation より

"Today, we’re releasing Qwen3.8-27B Dynamic v3.0 quants that deliver >10% top-1 better accuracy at the same size compared to every other provider."／"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1 accuracy yet being 89% smaller."／"UD-Q2_K_XL is around +8% more accurate on top-1 than the next best and it's 9.83GB and managed to create a working HTML program with 1 small JS bug - previously it would break."／"We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed"／"At all levels especially on the smaller quant sizes, Unsloth UD-3 quants get up to +10% extra top-1 accuracy at the same disk space!"／"All plots remove the MTP head from the x axis when calculating disk space to provide a fair comparison to everyone."／"We saw over 5.1 million Unsloth Qwen3.8 downloads in just 5 days!"（冒頭 ／ KL Divergence Benchmarks 節）— Unsloth Documentation より

"However top-1 is an argmax on 1 prediction, so it's not really effective on gauging actual inference."／"We created a dataset of 300 held out examples (NOT in calibration dataset) from Terminal-Bench 2.1 + DeepSWE + Harbor + MathArena 2025-26 + non-Latin/long-doc prompts and we did greedy argmax decoding for 32 tokens for BF16 vs all quants and providers."／"This allows us to gauge if there is overfitting and if quant outputs are similar to BF16's trajectories over multiple tokens. This is a better metric than top-1 accuracy since we extend KLD top-1 to more like KLD top-1 at 32 tokens."（Divergence-300 @32 節）— Unsloth Documentation より

"there is a sharp drop off from UD-Q2_K_XL to UD-IQ2_S for 32 token prediction from around 25% accuracy to under 8-10%. This sharp drop off means tool calling and non thinking modes break down."／"You will see a lot of looping when using quants below UD-Q2_K_XL - use presence_penalty = 1.5 in all cases (or higher)"／"Always enable thinking at least on low reasoning for 1-bit quants - non reasoning modes cause the model to not even output anyway"／"Do not use the model for tool calling - only general knowledge is retained when quantized heavily, and the model will either fail to call tools, keep calling tools or not even call them."／"A Top-1 recovery of 77% is not a replacement for Divergence-300 @32 at 8% which is a better indicator for actual inference workloads - you can use the model for very short general knowledge fact questions, but best to use UD-Q2_K_XL."（1-bit should not be used for agentic use-cases 節）— Unsloth Documentation より

"you can use the model for very short general knowledge fact questions, but best to use UD-Q2_K_XL."／"so we still use our old UD-2 for the larger quants - we plan to experiment and improve them as well!"（1-bit should not be used for agentic use-cases 節 ／ Not Overfitting 節）— Unsloth Documentation より