[Feature] Qwen3-30B-A3B-2507 AWQ的支持

[maxin9966]

Motivation

这个新模型我在官网网页上测试过，效果非常好，但是目前lmdeploy应该还没有对应的awq或gptq模型。

似乎vllm在sm75上不支持这个30B-A3B的gptq，不知道lmdeploy turbomind在qwen3-moe上对老架构显卡的兼容性如何？

Related resources

No response

Additional context

No response

[lzhangzz]

AWQ/GPTQ MoE models are supported since v0.6.2, and it works on all NVIDIA GPUs from sm70 to sm90.

[HuChundong]

可以正常跑，但是好像有中英文混乱的问题，使用ollama跑gguf没问题，不确定是lmdeploy的问题还是推理设置的参数问题。用的是swift的awq模型

[maxin9966]

QuantTrio/Qwen3-30B-A3B-Thinking-2507-AWQ 这个模型跑了报错。

日志:

Jul 31 07:30:07 ubuntu22lts systemd[1]: Started Uvicorn server for my app.

Jul 31 07:30:17 ubuntu22lts bash[1249758]: [102B blob data]
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/bin/lmdeploy", line 8, in
Jul 31 07:30:17 ubuntu22lts bash[1249758]: sys.exit(run())
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/cli/entrypoint.py", line 39, in run
Jul 31 07:30:17 ubuntu22lts bash[1249758]: args.run(args)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/cli/serve.py", line 371, in api_server
Jul 31 07:30:17 ubuntu22lts bash[1249758]: run_api_server(args.model_path,
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/serve/openai/api_server.py", line 1352, in serve
Jul 31 07:30:17 ubuntu22lts bash[1249758]: VariableInterface.async_engine = pipeline_class(model_path=model_path,
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/serve/async_engine.py", line 284, in init
Jul 31 07:30:17 ubuntu22lts bash[1249758]: self._build_turbomind(model_path=model_path, backend_config=backend_config, **kwargs)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/serve/async_engine.py", line 338, in _build_turbomind
Jul 31 07:30:17 ubuntu22lts bash[1249758]: self.engine = tm.TurboMind.from_pretrained(model_path,
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 386, in from_pretrained
Jul 31 07:30:17 ubuntu22lts bash[1249758]: return cls(model_path=pretrained_model_name_or_path,
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 159, in init
Jul 31 07:30:17 ubuntu22lts bash[1249758]: self._load_weights(model_source)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 178, in _load_weights
Jul 31 07:30:17 ubuntu22lts bash[1249758]: self._tm_model.export()
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/deploy/target_model/base.py", line 210, in export
Jul 31 07:30:17 ubuntu22lts bash[1249758]: for i, reader in self.input_model.readers():
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/deploy/source_model/llama.py", line 111, in readers
Jul 31 07:30:17 ubuntu22lts bash[1249758]: loader = create_loader(self.model_path, self.Reader.attn_layer_patten)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/deploy/loader.py", line 175, in create_loader
Jul 31 07:30:17 ubuntu22lts bash[1249758]: return SafetensorsLoader(*args, index_name=SAFE_WEIGHT_INDEX_NAME)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/deploy/loader.py", line 56, in init
Jul 31 07:30:17 ubuntu22lts bash[1249758]: self.shards, index = self.get_index(index_name, file_pattern)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/site-packages/lmdeploy/turbomind/deploy/loader.py", line 37, in get_index
Jul 31 07:30:17 ubuntu22lts bash[1249758]: index = json.load(f)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/json/init.py", line 293, in load
Jul 31 07:30:17 ubuntu22lts bash[1249758]: return loads(fp.read(),
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/json/init.py", line 346, in loads
Jul 31 07:30:17 ubuntu22lts bash[1249758]: return _default_decoder.decode(s)
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/json/decoder.py", line 337, in decode
Jul 31 07:30:17 ubuntu22lts bash[1249758]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Jul 31 07:30:17 ubuntu22lts bash[1249758]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 31 07:30:17 ubuntu22lts bash[1249758]: File "/home/ma/miniconda3/envs/lmdeploy3/lib/python3.11/json/decoder.py", line 355, in raw_decode
Jul 31 07:30:17 ubuntu22lts bash[1249758]: raise JSONDecodeError("Expecting value", s, err.value) from None
Jul 31 07:30:17 ubuntu22lts bash[1249758]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Jul 31 07:30:19 ubuntu22lts bash[1249758]: [68B blob data]
Jul 31 07:30:19 ubuntu22lts systemd[1]: lm-qwen3-30b-a3b.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 07:30:19 ubuntu22lts systemd[1]: lm-qwen3-30b-a3b.service: Failed with result 'exit-code'.
Jul 31 07:30:19 ubuntu22lts systemd[1]: lm-qwen3-30b-a3b.service: Consumed 13.713s CPU time.

[BUJIDAOVS]

希望支持https://hf-mirror.com/cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ

目前0.9.2报错：
lmdeploy-openai_1 |
lmdeploy-openai_1 | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
lmdeploy-openai_1 |
lmdeploy-openai_1 | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
lmdeploy-openai_1 | By pulling and using the container, you accept the terms and conditions of this license:
lmdeploy-openai_1 | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
lmdeploy-openai_1 |
lmdeploy-openai_1 | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
lmdeploy-openai_1 |
lmdeploy-openai_1 | Traceback (most recent call last):
lmdeploy-openai_1 | File "/opt/py3/bin/lmdeploy", line 33, in
lmdeploy-openai_1 | sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/cli/entrypoint.py", line 39, in run
lmdeploy-openai_1 | args.run(args)
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/cli/serve.py", line 360, in api_server
lmdeploy-openai_1 | run_api_server(args.model_path,
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/serve/openai/api_server.py", line 1323, in serve
lmdeploy-openai_1 | VariableInterface.async_engine = pipeline_class(model_path=model_path,
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 281, in init
lmdeploy-openai_1 | self._build_turbomind(model_path=model_path, backend_config=backend_config, **kwargs)
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 332, in _build_turbomind
lmdeploy-openai_1 | self.engine = tm.TurboMind.from_pretrained(model_path,
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 381, in from_pretrained
lmdeploy-openai_1 | return cls(model_path=pretrained_model_name_or_path,
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 154, in init
lmdeploy-openai_1 | self.model_comm = self._from_hf(model_source=model_source,
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/turbomind/turbomind.py", line 271, in _from_hf
lmdeploy-openai_1 | tm_model = get_tm_model(model_path, self.model_name, self.chat_template_name, engine_config)
lmdeploy-openai_1 | File "/opt/lmdeploy/lmdeploy/turbomind/deploy/converter.py", line 206, in get_tm_model
lmdeploy-openai_1 | assert 0, f'unsupported quant_config: {quant_config}'
lmdeploy-openai_1 | AssertionError: unsupported quant_config: {'config_groups': {'group_0': {'input_activations': None, 'output_activations': None, 'targets': ['Linear'], 'weights': {'actorder': None, 'block_structure': None, 'dynamic': False, 'group_size': 128, 'num_bits': 4, 'observer': 'minmax', 'observer_kwargs': {}, 'strategy': 'group', 'symmetric': True, 'type': 'int'}}}, 'format': 'pack-quantized', 'global_compression_ratio': None, 'ignore': ['model.layers.0.mlp.gate', 'model.layers.1.mlp.gate', 'model.layers.2.mlp.gate', 'model.layers.3.mlp.gate', 'model.layers.4.mlp.gate', 'model.layers.5.mlp.gate', 'model.layers.6.mlp.gate', 'model.layers.7.mlp.gate', 'model.layers.8.mlp.gate', 'model.layers.9.mlp.gate', 'model.layers.10.mlp.gate', 'model.layers.11.mlp.gate', 'model.layers.12.mlp.gate', 'model.layers.13.mlp.gate', 'model.layers.14.mlp.gate', 'model.layers.15.mlp.gate', 'model.layers.16.mlp.gate', 'model.layers.17.mlp.gate', 'model.layers.18.mlp.gate', 'model.layers.19.mlp.gate', 'model.layers.20.mlp.gate', 'model.layers.21.mlp.gate', 'model.layers.22.mlp.gate', 'model.layers.23.mlp.gate', 'model.layers.24.mlp.gate', 'model.layers.25.mlp.gate', 'model.layers.26.mlp.gate', 'model.layers.27.mlp.gate', 'model.layers.28.mlp.gate', 'model.layers.29.mlp.gate', 'model.layers.30.mlp.gate', 'model.layers.31.mlp.gate', 'model.layers.32.mlp.gate', 'model.layers.33.mlp.gate', 'model.layers.34.mlp.gate', 'model.layers.35.mlp.gate', 'model.layers.36.mlp.gate', 'model.layers.37.mlp.gate', 'model.layers.38.mlp.gate', 'model.layers.39.mlp.gate', 'model.layers.40.mlp.gate', 'model.layers.41.mlp.gate', 'model.layers.42.mlp.gate', 'model.layers.43.mlp.gate', 'model.layers.44.mlp.gate', 'model.layers.45.mlp.gate', 'model.layers.46.mlp.gate', 'model.layers.47.mlp.gate', 'lm_head'], 'kv_cache_scheme': None, 'quant_method': 'compressed-tensors', 'quantization_status': 'compressed'}

[lzhangzz]

@maxin9966 测了

Qwen/Qwen3-30B-A3B-GPTQ-Int4
QuantTrio/Qwen3-30B-A3B-Thinking-2507-AWQ

都没有问题

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

你的 log 报错的地方是个 JSON parse error，似乎是读 model.safetensors.index.json 的时候报的，建议检查下模型的完整性

[lzhangzz]

@BUJIDAOVS cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ

这个 quant config 和一般 AWQ/GPTQ 的区别有点大，暂时没有人力来支持。可以试试 QuantTrio/Qwen3-30B-A3B-Thinking-2507-AWQ