[Bug]: RuntimeError: setup failed!

[ShuhangChen1207]

Your current environment

absl-py 2.1.0
accelerate 1.5.2
addict 2.4.0
aiofiles 23.2.1
aiohappyeyeballs 2.6.1
aiohttp 3.11.14
aiosignal 1.3.2
airportsdata 20250224
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-types 0.7.0
antlr4-python3-runtime 4.13.2
anyio 4.9.0
astor 0.8.1
async-timeout 5.0.1
attrdict 2.0.1
attrs 25.1.0
auto_tune 0.1.0
binpacking 1.5.2
blake3 1.0.4
certifi 2025.1.31
cffi 1.17.1
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
compressed-tensors 0.9.1
contourpy 1.3.1
cpm-kernels 1.0.11
crcmod 1.7
cryptography 44.0.2
cycler 0.12.1
Cython 3.0.12
dacite 1.9.2
dataflow 0.0.1
datasets 3.2.0
decorator 5.1.1
deepspeed 0.14.5
depyf 0.18.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.7.0
einops 0.8.1
email_validator 2.2.0
exceptiongroup 1.2.2
fastapi 0.115.12
fastapi-cli 0.0.7
ffmpy 0.5.0
filelock 3.18.0
fonttools 4.56.0
frozenlist 1.5.0
fsspec 2024.9.0
future 1.0.0
gguf 0.10.0
gradio 5.23.0
gradio_client 1.8.0
groovy 0.1.2
grpcio 1.71.0
h11 0.14.0
hccl 0.1.0
hccl_parser 0.1
hjson 3.1.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.29.3
idna 3.10
importlib_metadata 8.6.1
interegular 0.3.3
jieba 0.42.1
Jinja2 3.1.6
jiter 0.9.0
jmespath 0.10.0
joblib 1.4.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
kiwisolver 1.4.8
lark 1.2.2
latex2sympy2_extended 1.10.1
llm_datadist 0.0.1
llvmlite 0.43.0
lm-format-enforcer 0.10.11
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
math-verify 0.7.0
matplotlib 3.10.1
mdurl 0.1.2
mistral_common 1.5.4
ml_dtypes 0.5.1
modelscope 1.24.0
mpmath 1.3.0
ms_swift 3.2.1
msgspec 0.19.0
msobjdump 0.1.0
multidict 6.2.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.4
nltk 3.9.1
numba 0.60.0
numpy 1.26.4
nvidia-ml-py 12.570.86
op_compile_tool 0.1.0
op_gen 0.1
op_test_frame 0.1
opc_tool 0.1.0
openai 1.68.2
opencv-python-headless 4.11.0.86
orjson 3.10.16
oss2 2.19.1
outlines 0.1.11
outlines_core 0.1.26
packaging 24.2
pandas 2.2.3
partial-json-parser 0.2.1.1.post5
pathlib2 2.3.7.post1
peft 0.14.0
pillow 11.1.0
pip 25.0.1
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache 0.3.0
protobuf 3.20.0
psutil 7.0.0
py-cpuinfo 9.0.0
pyarrow 19.0.1
pycountry 24.6.1
pycparser 2.22
pycryptodome 3.22.0
pydantic 2.10.6
pydantic_core 2.27.2
pydub 0.25.1
Pygments 2.19.1
pyparsing 3.2.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.20
pytz 2025.1
PyYAML 6.0.2
pyzmq 26.3.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 13.9.4
rich-toolkit 0.13.2
rouge 1.0.1
rpds-py 0.23.1
ruff 0.11.2
safehttpx 0.1.6
safetensors 0.5.3
schedule_search 0.0.1
scipy 1.15.2
semantic-version 2.10.0
sentencepiece 0.2.0
setuptools 69.5.1
setuptools-scm 8.2.0
shellingham 1.5.4
show_kernel_debug_data 0.1.0
simplejson 3.20.1
six 1.17.0
sniffio 1.3.1
sortedcontainers 2.4.0
starlette 0.46.1
sympy 1.13.1
te 0.4.0
tensorboard 2.19.0
tensorboard-data-server 0.7.2
tiktoken 0.9.0
tokenizers 0.21.1
tomli 2.2.1
tomlkit 0.13.2
torch 2.5.1
torch-npu 2.5.1.dev20250308
torchaudio 2.5.1
torchvision 0.20.1
tornado 6.4.2
tqdm 4.67.1
transformers 4.50.0
transformers-stream-generator 0.0.5
trl 0.16.0
typer 0.15.2
typing_extensions 4.12.2
tzdata 2025.2
urllib3 2.3.0
uvicorn 0.34.0
uvloop 0.21.0
vllm 0.7.3
vllm_ascend 0.7.3rc1
watchfiles 1.0.4
websockets 15.0.1
Werkzeug 3.1.3
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
zstandard 0.23.0

🐛 Describe the bug

{'loss': 3.14e-06, 'grad_norm': 0.00347595, 'learning_rate': 3e-08, 'train_speed(iter/s)': 0.028684, 'completion_length': 74.35416985, 'response_clip_ratio': 0.0, 'rewards/MathFormat': 0.0, 'reward': 0.0, 'reward_std': 0.0, 'kl': 0.00330734, 'clip_ratio': 0.0, 'epoch': 0.0, 'global_step/max_steps': '5/8724', 'percentage': '0.06%', 'elapsed_time': '2m 25s', 'remaining_time': '2d 22h 37m 44s'}
Train: 0%| | 5/8724 [02:25<69:44:50, 28.80s/it]mki_log log dir:/root/atb/log exist
[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/cli/rlhf.py", line 5, in
[rank0]: rlhf_main()
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/llm/train/rlhf.py", line 96, in rlhf_main
[rank0]: return SwiftRLHF(args).main()
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/llm/base.py", line 47, in main
[rank0]: result = self.run()
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/llm/train/sft.py", line 142, in run
[rank0]: return self.train(trainer)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/llm/train/sft.py", line 204, in train
[rank0]: trainer.train(trainer.args.resume_from_checkpoint)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/trainers/mixin.py", line 288, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2245, in train
[rank0]: return inner_training_loop(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/trainer.py", line 2556, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/trainer.py", line 3712, in training_step
[rank0]: inputs = self._prepare_inputs(inputs)
[rank0]: File "/workspace/trl/extras/profiling.py", line 87, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: File "/workspace/trl/trainer/grpo_trainer.py", line 692, in _prepare_inputs
[rank0]: inputs = self._generate_and_score_completions(inputs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 700, in _generate_and_score_completions
[rank0]: inputs, outputs = self._fast_infer(inputs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 659, in _fast_infer
[rank0]: outputs = self.engine.infer(_input_slice, self.request_config, use_tqdm=False)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 422, in infer
[rank0]: step_outputs = self.engine.step()
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1391, in step
[rank0]: outputs = self.model_executor.execute_model(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 139, in execute_model
[rank0]: output = self.collective_rpc("execute_model",
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]: answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
[rank0]: output = self.model_runner.execute_model(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/worker/model_runner.py", line 1140, in execute_model
[rank0]: hidden_or_intermediate_states = model_executable(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 486, in forward
[rank0]: hidden_states = self.model(input_ids, positions, kv_caches,
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 172, in call
[rank0]: return self.forward(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 348, in forward
[rank0]: hidden_states, residual = layer(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 247, in forward
[rank0]: hidden_states = self.self_attn(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 179, in forward
[rank0]: attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/attention/layer.py", line 198, in forward
[rank0]: return self.impl.forward(self, query, key, value,
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/attention.py", line 869, in forward
[rank0]: torch_npu._npu_flash_attention_qlens(
[rank0]: File "/usr/local/python3.10/lib/python3.10/site-packages/torch/_ops.py", line 1116, in call
[rank0]: return self._op(*args, **(kwargs or {}))
[rank0]: RuntimeError: setup failed!

[wulitaotao1]

相同的问题

环境：
openEuler release 24.03 (LTS)
NAME="openEuler"
VERSION="24.03 (LTS)"
ID="openEuler"
VERSION_ID="24.03"
PRETTY_NAME="openEuler 24.03 (LTS)"
ANSI_COLOR="0;31"

Linux resource-group-01 5.15.0-25-generic modelscope/ms-swift#25 SMP Wed Mar 30 15:57:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

glibc-2.38-47.oe2403.aarch64
glibc-common-2.38-47.oe2403.aarch64
glibc-devel-2.38-47.oe2403.aarch64

Ascend-hdk-910b-npu-firmware 7.5.0.5.220
Ascend-hdk-910b-npu-driver 24.1.0.3
CANN 8.1.RC1
torch 2.5.1
torch-npu 2.5.1.dev20250308
vllm 0.7.3+empty
vllm_ascend 0.7.3rc1
trl 0.16.0
ms_swift 3.3.0.dev0

训练中报错：
`[rank0]:[E326 08:57:33.451117940 compiler_depend.ts:418] setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:129 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffffadcec908 in /usr/local/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame modelscope/ms-swift#1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x70 (0xffffadc9b4e0 in /usr/local/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame modelscope/ms-swift#2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x8c (0xfffec03944ac in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame modelscope/ms-swift#3: + 0x34adc (0xfffec0394adc in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame modelscope/ms-swift#4: + 0x159bca4 (0xffffa126bca4 in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame modelscope/ms-swift#5: + 0x743d04 (0xffffa0413d04 in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame modelscope/ms-swift#6: + 0x744518 (0xffffa0414518 in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame modelscope/ms-swift#7: + 0x74129c (0xffffa041129c in /usr/local/lib64/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame modelscope/ms-swift#8: + 0x4c9e4c (0xffffadd29e4c in /usr/local/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame modelscope/ms-swift#9: + 0x80104 (0xffffb8760104 in /usr/lib64/libc.so.6)
frame modelscope/ms-swift#10: + 0xe7c9c (0xffffb87c7c9c in /usr/lib64/libc.so.6)

[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 5, in
[rank0]: rlhf_main()
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 98, in rlhf_main
[rank0]: return SwiftRLHF(args).main()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 47, in main
[rank0]: result = self.run()
[rank0]: ^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 142, in run
[rank0]: return self.train(trainer)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 202, in train
[rank0]: trainer.train(trainer.args.resume_from_checkpoint)
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 288, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1042, in training_step
[rank0]: return super().training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 3692, in training_step
[rank0]: inputs = self._prepare_inputs(inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/trl/extras/profiling.py", line 87, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 647, in _prepare_inputs
[rank0]: inputs = self._generate_and_score_completions(inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 814, in _generate_and_score_completions
[rank0]: inputs, outputs = self._fast_infer(inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 772, in _fast_infer
[rank0]: outputs = self._infer_multi_turn(_input_slice, self.request_config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 636, in _infer_multi_turn
[rank0]: results: List[ChatCompletionResponse] = self.engine.infer(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 432, in infer
[rank0]: step_outputs = self.engine.step()
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1391, in step
[rank0]: outputs = self.model_executor.execute_model(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 139, in execute_model
[rank0]: output = self.collective_rpc("execute_model",
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]: answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/utils.py", line 2196, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
[rank0]: output = self.model_runner.execute_model(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm_ascend/worker/model_runner.py", line 1140, in execute_model
[rank0]: hidden_or_intermediate_states = model_executable(
[rank0]: ^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 486, in forward
[rank0]: hidden_states = self.model(input_ids, positions, kv_caches,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 172, in call
[rank0]: return self.forward(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 348, in forward
[rank0]: hidden_states, residual = layer(
[rank0]: ^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 247, in forward
[rank0]: hidden_states = self.self_attn(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 179, in forward
[rank0]: attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm/attention/layer.py", line 198, in forward
[rank0]: return self.impl.forward(self, query, key, value,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/vllm_ascend/attention.py", line 801, in forward
[rank0]: torch_npu._npu_reshape_and_cache(key=key,
[rank0]: File "/usr/local/lib64/python3.11/site-packages/torch/_ops.py", line 1116, in call
[rank0]: return self._op(*args, **(kwargs or {}))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[rank0]: [ERROR] 2025-03-26-08:57:33 (PID:931703, Device:7, RankID:0) ERR00100 PTA call acl api failed.`

[Yikun]

@ji-huazhong Do you have any idea about this?

[arcyleung]

Any follow up or clues about this problem? I am also trying to deploy deepseek-r1 bf16 on 32 x 910B3 using the docker image ascend-vllm:v0.7.3-dev 5aabb7c89fa2.

I have CANN 8.0.0.beta1 packages installed

The checkpoints load successfully but calling inference on the v1/completions causes the "setup failed!" crash
deepseek_npu_crash.txt

Drivers and versions

 /usr/local/Ascend/driver/tools/upgrade-tool --device_index -1 --system_version
{
Get system version(24.1.0) succeed, deviceId(0)
        {"device_id":0, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(1)
        {"device_id":1, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(2)
        {"device_id":2, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(3)
        {"device_id":3, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(4)
        {"device_id":4, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(5)
        {"device_id":5, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(6)
        {"device_id":6, "version":24.1.0}
Get system version(24.1.0) succeed, deviceId(7)
        {"device_id":7, "version":24.1.0}

[Yikun]

@MengqingCao @wangxiyuan Would mind taking a look on this issue? Looks like it happened multiple times. My guess is that it may be related to some configuration or environment variables?

[quanliu1991]

@Yikun The same problem, Sometimes inference succeeds, sometimes inference errors are reported.

docker images vision 0.7.3.rc1

uname -a:
Linux 5f84881fa56f 5.10.0-60.18.0.50.oe2203.aarch64 #1 SMP Wed Mar 30 02:43:08 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

device: 910B3

INFO 04-10 11:26:01 engine.py:280] Added request chatcmpl-fae80b33fa5f4ee4a4d227e2eae39efc.
mki_log mkdir /root/atb/
mki_log mkdir /root/atb/log
mki_log log dir:/root/atb/log exist
[rank0]:[E410 11:26:01.716153860 compiler_depend.ts:418] setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:129 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffff853ac908 in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x70 (0xffff8535b4e0 in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x8c (0xfffd1e7344ac in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: + 0x34adc (0xfffd1e734adc in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: + 0x159d034 (0xfffddd4ad034 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: + 0x743db4 (0xfffddc653db4 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: + 0x7445c8 (0xfffddc6545c8 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: + 0x74134c (0xfffddc65134c in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: + 0x4c9e4c (0xffff853e9e4c in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: + 0x7d5b8 (0xffff8fddd5b8 in /lib/aarch64-linux-gnu/libc.so.6)
frame #10: + 0xe5edc (0xffff8fe45edc in /lib/aarch64-linux-gnu/libc.so.6)

[rank1]:[E410 11:26:01.716641274 compiler_depend.ts:418] setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:129 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffff853ac908 in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x70 (0xffff8535b4e0 in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x8c (0xfffd2c0444ac in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: + 0x34adc (0xfffd2c044adc in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: + 0x159d034 (0xfffddd4ad034 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: + 0x743db4 (0xfffddc653db4 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: + 0x7445c8 (0xfffddc6545c8 in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: + 0x74134c (0xfffddc65134c in /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: + 0x4c9e4c (0xffff853e9e4c in /usr/local/python3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: + 0x7d5b8 (0xffff8fddd5b8 in /lib/aarch64-linux-gnu/libc.so.6)
frame #10: + 0xe5edc (0xffff8fe45edc in /lib/aarch64-linux-gnu/libc.so.6)

ERROR 04-10 11:26:01 engine.py:140] RuntimeError('The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.\nSince the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.\nNote: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.\n[ERROR] 2025-04-10-11:26:01 (PID:294, Device:0, RankID:-1) ERR00100 PTA call acl api failed.\n')
ERROR 04-10 11:26:01 engine.py:140] Traceback (most recent call last):
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 138, in start
ERROR 04-10 11:26:01 engine.py:140] self.run_engine_loop()
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 201, in run_engine_loop
ERROR 04-10 11:26:01 engine.py:140] request_outputs = self.engine_step()
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 219, in engine_step
ERROR 04-10 11:26:01 engine.py:140] raise e
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 210, in engine_step
ERROR 04-10 11:26:01 engine.py:140] return self.engine.step()
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1391, in step
ERROR 04-10 11:26:01 engine.py:140] outputs = self.model_executor.execute_model(
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 284, in execute_model
ERROR 04-10 11:26:01 engine.py:140] driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
ERROR 04-10 11:26:01 engine.py:140] return self.driver_worker.execute_model(execute_model_req)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
ERROR 04-10 11:26:01 engine.py:140] output = self.model_runner.execute_model(
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 04-10 11:26:01 engine.py:140] return func(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/worker/model_runner.py", line 1140, in execute_model
ERROR 04-10 11:26:01 engine.py:140] hidden_or_intermediate_states = model_executable(
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-10 11:26:01 engine.py:140] return self._call_impl(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-10 11:26:01 engine.py:140] return forward_call(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 492, in forward
ERROR 04-10 11:26:01 engine.py:140] hidden_states = self.model(input_ids, positions, kv_caches,
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 172, in call
ERROR 04-10 11:26:01 engine.py:140] return self.forward(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 350, in forward
ERROR 04-10 11:26:01 engine.py:140] hidden_states, residual = layer(
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-10 11:26:01 engine.py:140] return self._call_impl(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-10 11:26:01 engine.py:140] return forward_call(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 247, in forward
ERROR 04-10 11:26:01 engine.py:140] hidden_states, residual = self.input_layernorm(
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-10 11:26:01 engine.py:140] return self._call_impl(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-10 11:26:01 engine.py:140] return forward_call(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/custom_op.py", line 25, in forward
ERROR 04-10 11:26:01 engine.py:140] return self._forward_method(*args, **kwargs)
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/ops/layernorm.py", line 32, in forward_oot
ERROR 04-10 11:26:01 engine.py:140] x, _, residual = torch_npu.npu_add_rms_norm(x, residual, self.weight,
ERROR 04-10 11:26:01 engine.py:140] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/_ops.py", line 1116, in call
ERROR 04-10 11:26:01 engine.py:140] return self._op(*args, **(kwargs or {}))
ERROR 04-10 11:26:01 engine.py:140] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
ERROR 04-10 11:26:01 engine.py:140] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
ERROR 04-10 11:26:01 engine.py:140] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
ERROR 04-10 11:26:01 engine.py:140] [ERROR] 2025-04-10-11:26:01 (PID:294, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
ERROR 04-10 11:26:01 engine.py:140]
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] Traceback (most recent call last):
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 236, in _run_worker_process
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] output = self.model_runner.execute_model(
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm_ascend/worker/model_runner.py", line 1140, in execute_model
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 492, in forward
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 172, in call
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return self.forward(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 350, in forward
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] hidden_states, residual = layer(
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 259, in forward
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] hidden_states = self.mlp(hidden_states)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 99, in forward
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] x, _ = self.down_proj(x)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 1149, in forward
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] output_parallel = self.quant_method.apply(self,
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] File "/usr/local/python3.10/lib/python3.10/site-packages/vllm/model_executor/layers/linear.py", line 142, in apply
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] return F.linear(x, layer.weight, bias)
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242] [ERROR] 2025-04-10-11:26:01 (PID:770, Device:1, RankID:-1) ERR00100 PTA call acl api failed.
(VllmWorkerProcess pid=770) ERROR 04-10 11:26:01 multiproc_worker_utils.py:242]
CRITICAL 04-10 11:26:01 launcher.py:104] MQLLMEngine is already dead, terminating server process

[quanliu1991]

@MengqingCao @wangxiyuan Would mind taking a look on this issue? Looks like it happened multiple times. My guess is that it may be related to some configuration or environment variables?

I guess it might be the prefix caching feature that causes the SelfAttentionOperation operator to work incorrectly, because this problem does not occur after turning off the prefix caching feature.

[wangxiyuan]

if you are using prefix caching, there is an known issue #447 . It'll be fixed when CANN8.1 nnal package is released.

[wangxiyuan]

No response for long time. Close this one now. Feel free to reopen a new issue if it's still a problem. Thanks.