unsloth

/

DeepSeek-R1-Distill-Qwen-32B-bnb-4bit

like 29

Follow

Unsloth AI 28.3k

I ran the VLLM service command the same as the document example as well as in the discussion but get error

INFO 01-31 07:17:38 loader.py:1059] Loading weights with BitsAndBytes quantization. May take a while ...
INFO 01-31 07:17:38 weight_utils.py:251] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]

ERROR 01-31 07:17:39 engine.py:387] Blockwise quantization only supports 16/32-bit floats, but got torch.uint8
ERROR 01-31 07:17:39 engine.py:387] Traceback (most recent call last):
ERROR 01-31 07:17:39 engine.py:387] File "/root/miniconda3/envs/Chloe/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
ERROR 01-31 07:17:39 engine.py:387] engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 01-31 07:17:39 engine.py:387] File "/root/miniconda3/envs/Chloe/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 121, in from_engine_args
ERROR 01-31 07:17:39 engine.py:387] return cls(ipc_path=ipc_path,

This is the command I ran
vllm serve
"unsloth/DeepSeek-R1-Distill-Qwen-32B-bnb-4bit"
--load-format bitsandbytes
--quantization bitsandbytes
--max-model-len=8192
--gpu-memory-utilization=0.99
--host 0.0.0.0
--port 8000
oh weird the bnb version should work since it's just qwen, could you file a ticket on their github issue?

So that means I should could ran the VLLM with this bnb-4bit model?

I ran into this problem with the Llama-70B-bnb-4bit model.

I think the problem is that there's no quantization_config key in the config.json, so vllm tries to requantize the model weights, leading to this error. To stop the error, I manually added the quantization_config from the previous distillation to config.json that was downloaded.

For Llama-70B-bnb-4bit, I copied it from here

"quantization_config": {
    "_load_in_4bit": true,
    "_load_in_8bit": false,
    "bnb_4bit_compute_dtype": "bfloat16",
    "bnb_4bit_quant_storage": "uint8",
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_use_double_quant": true,
    "llm_int8_enable_fp32_cpu_offload": false,
    "llm_int8_has_fp16_weight": false,
    "llm_int8_skip_modules": [
      "lm_head",
      "multi_modal_projector",
      "merger",
      "modality_projection"
    ],
    "llm_int8_threshold": 6.0,
    "load_in_4bit": true,
    "load_in_8bit": false,
    "quant_method": "bitsandbytes"
  },

For Qwen-32B-bnb-4bit, I would think it's this one:

"quantization_config": {
    "_load_in_4bit": true,
    "_load_in_8bit": false,
    "bnb_4bit_compute_dtype": "bfloat16",
    "bnb_4bit_quant_storage": "uint8",
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_use_double_quant": true,
    "llm_int8_enable_fp32_cpu_offload": false,
    "llm_int8_has_fp16_weight": false,
    "llm_int8_skip_modules": null,
    "llm_int8_threshold": 6.0,
    "load_in_4bit": true,
    "load_in_8bit": false,
    "quant_method": "bitsandbytes"
  },

All that said, I'm not sure if the same quantization configs were used for the distilled models though, so it would be great if someone from unsloth could confirm. @shimmyshimmer

Yes, that would be great if someone from unsloth could confirm. : ) I get ERROR 02-01 22:35:23 engine.py:387] start (0) + length (6912) exceeds dimension size (1). after adding "quantization_config": {
"_load_in_4bit": true,
"_load_in_8bit": false,
"bnb_4bit_compute_dtype": "bfloat16",
"bnb_4bit_quant_storage": "uint8",
"bnb_4bit_quant_type": "nf4",
"bnb_4bit_use_double_quant": true,
"llm_int8_enable_fp32_cpu_offload": false,
"llm_int8_has_fp16_weight": false,
"llm_int8_skip_modules": null,
"llm_int8_threshold": 6.0,
"load_in_4bit": true,
"load_in_8bit": false,
"quant_method": "bitsandbytes"
}, in config.json

I ran into this problem with the Llama-70B-bnb-4bit model.

I think the problem is that there's no quantization_config key in the config.json, so vllm tries to requantize the model weights, leading to this error. To stop the error, I manually added the quantization_config from the previous distillation to config.json that was downloaded.

For Llama-70B-bnb-4bit, I copied it from here

"quantization_config": {
    "_load_in_4bit": true,
    "_load_in_8bit": false,
    "bnb_4bit_compute_dtype": "bfloat16",
    "bnb_4bit_quant_storage": "uint8",
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_use_double_quant": true,
    "llm_int8_enable_fp32_cpu_offload": false,
    "llm_int8_has_fp16_weight": false,
    "llm_int8_skip_modules": [
      "lm_head",
      "multi_modal_projector",
      "merger",
      "modality_projection"
    ],
    "llm_int8_threshold": 6.0,
    "load_in_4bit": true,
    "load_in_8bit": false,
    "quant_method": "bitsandbytes"
  },

For Qwen-32B-bnb-4bit, I would think it's this one:

"quantization_config": {
    "_load_in_4bit": true,
    "_load_in_8bit": false,
    "bnb_4bit_compute_dtype": "bfloat16",
    "bnb_4bit_quant_storage": "uint8",
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_use_double_quant": true,
    "llm_int8_enable_fp32_cpu_offload": false,
    "llm_int8_has_fp16_weight": false,
    "llm_int8_skip_modules": null,
    "llm_int8_threshold": 6.0,
    "load_in_4bit": true,
    "load_in_8bit": false,
    "quant_method": "bitsandbytes"
  },

All that said, I'm not sure if the same quantization configs were used for the distilled models though, so it would be great if someone from unsloth could confirm. @shimmyshimmer

Did you revise any other files or only config.json?

@ChloeHuang1 @blnewman-uw thanks a lot guys we fixed the problem and reuploaded the models so good catch finding it out! :)