Why is the bitsandbytes model significantly slower than the AWQ model?

[hahmad2008]

Your current environment

VLLM 0.6.1.post2

🐛 Describe the bug

I used a model from a hub with AWQ quantization, so it's already quantized. I loaded it with a half data type, and it performs really fast. However, when I loaded the base model and let VLLM handle bitsandbytes quantization, the performance was significantly slower compared to the AWQ model imported directly from the hub.
Any idea?

[hmellor]

The main two reasons I can think of are:

• AWQ has fast custom kernels in https://github.com/vllm-project/vllm/tree/main/csrc/quantization/awq
• bnb is not compatible with CUDA graph yet, whereas AWQ is

How do they compare if you run both with --enforce-eager?

[mgoin]

@hahmad2008 can you see if this PR fixes this issue? #8768

[hahmad2008]

@mgoin Thanks for mentioning this.
I tried to build it from source an I got this error:

Obtaining file:///root/vllm
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "/root/miniconda3/envs/vllm-support-bnb-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/miniconda3/envs/vllm-support-bnb-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/miniconda3/envs/vllm-support-bnb-env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
        File "/tmp/pip-build-env-vxibqh_4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 464, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
        File "/tmp/pip-build-env-vxibqh_4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-vxibqh_4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-vxibqh_4/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
        File "<string>", line 12, in <module>
      ModuleNotFoundError: No module named 'torch'
      [end of output]

what I did:

• conda install pytorch torchvision pytorch-cuda -c pytorch -c nvidia
• git clone https://github.com/jeejeelee/vllm.git
• cd vllm
• git checkout bitsandbytes-branch
• pip install -e .

[jeejeelee]

@hahmad2008 Now, you can try the latest version 0.6.2 rather than build from source.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!