[qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0)

[TerryT9]

Name and Version

register_backend: registered backend QNN (2 devices) register_device: registered device qnn-npu (Qualcomm NPU) register_device: registered device qnn-gpu (Qualcomm Adreno GPU) register_backend: registered backend CPU (1 devices) register_device: registered device CPU (CPU) version: 4617 (5f93376f) built with for x86_64-w64-windows-gnu [ggml_backend_qnn_reg_impl, 466]: qnn backend registry init

Operating systems

Windows, Other? (Please let us know in description)

GGML backends

CPU

Hardware

Qualcomm Adreno GPU
Snapdragon 8+ Gen 1

Models

• model: Llama3-1B.f16

Problem description & steps to reproduce

I got a lot of error meesages: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal

[device_supports_op, 482]: [VIEW]unsupported op [device_supports_op, 482]: [VIEW]unsupported op [device_supports_op, 482]: [CPY]unsupported op [device_supports_op, 482]: [CPY]unsupported op [device_supports_op, 482]: [VIEW]unsupported op [device_supports_op, 482]: [VIEW]unsupported op [device_supports_op, 482]: [VIEW]unsupported op [device_supports_op, 482]: [VIEW]unsupported op [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/343 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/365 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/344 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/366 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/345 [ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/367 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/346 [ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/368 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/347 [ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/369 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/348 [ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/370 [ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/349 [ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/371

First Bad Commit

No response

Relevant log output

register_backend: registered backend QNN (2 devices)
register_device: registered device qnn-npu (Qualcomm NPU)
register_device: registered device qnn-gpu (Qualcomm Adreno GPU)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
build: 4617 (5f93376f) with  for x86_64-w64-windows-gnu (debug)
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file: using device qnn-gpu (Qualcomm Adreno GPU) - 7672 MiB free
llama_model_loader: loaded meta data with 22 key-value pairs and 201 tensors from /data/local/tmp/fp16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = TinyLlama
llama_model_loader: - kv   2:                       llama.context_length u32              = 2048
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv   4:                          llama.block_count u32              = 22
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 5632
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 64
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 4
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 1
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                      tokenizer.ggml.merges arr[str,61249]   = ["▁ t", "e r", "i n", "▁ a", "e n...
llama_model_loader: - kv  17:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  18:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  19:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  20:            tokenizer.ggml.padding_token_id u32              = 2
llama_model_loader: - kv  21:                    tokenizer.chat_template str              = {% for message in messages %}\n{% if m...
llama_model_loader: - type  f32:   45 tensors
llama_model_loader: - type  f16:  156 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 3
llm_load_vocab: token to piece cache size = 0.1684 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
[ggml_backend_qnn_reg_impl, 466]: qnn backend registry init
[ggml_backend_qnn_device_get_memory, 297]: free memory: 7672MB, total memory: 32627MB
[ggml_backend_qnn_device_get_memory, 297]: free memory: 7670MB, total memory: 32627MB
[device_supports_op, 482]: [GET_ROWS]unsupported op
[ggml_qnn_supports_matmul_op, 442]: [qnn-npu][MUL_MAT]tensor size is too large, support/unsupported: 0/1
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/2
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/3
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/4
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/5
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/6
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/7
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/8
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/9
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/10
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/11
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/12
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/13
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/14
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/15
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/16
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/17
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/18
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/19
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/20
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/21
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/22
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/23
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/24
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/25
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/26
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/27
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/28
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/29
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/30
[ggml_qnn_supports_matmul_op, 45llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 2048
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 22
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 64
llm_load_print_meta: n_embd_head_v    = 64
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 256
llm_load_print_meta: n_embd_v_gqa     = 256
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 5632
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 2048
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
2]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/31
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/32
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/33
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/34
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/35
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/36
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/37
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/38
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/39
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/40
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/41
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/42
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/43
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/44
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/45
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/46
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/47
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/48
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/49
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/50
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/51
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/52
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/53
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/54
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/55
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/56
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/57
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/58
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/59
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/60
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/61
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/62
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 1B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 1.10 B
llm_load_print_meta: model size       = 2.05 GiB (16.00 BPW)
llm_load_print_meta: general.name     = TinyLlama
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 2 '</s>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_print_meta: EOG token        = 2 '</s>'
llm_load_print_meta: max token length = 48
1), src1(0) and op(0) are not equal, support/unsupported: 0/63
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/64
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/65
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/66
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/67
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/68
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/69
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/70
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/71
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/72
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/73
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/74
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/75
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/76
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/77
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/78
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/79
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/80
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/81
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/82
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/83
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/84
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/85
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/86
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/87
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/88
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/89
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/90
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/91
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/92
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/93
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/94
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/95
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/96
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/97
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/98
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/99
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/100
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/101
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/102
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/103
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/104
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/105
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/106
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/107
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/108
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/109
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/110
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/111
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/112
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/113
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/114
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/115
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/116
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/117
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/118
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/119
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/120
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/121
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/122
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/123
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/124
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/125
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/126
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/23 layers to GPU
llm_load_tensors:   CPU_Mapped model buffer size =  2098.35 MiB
..........................................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 10000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_pre_seq (4096) > n_ctx_train (2048) -- possible training context overflow
support/unsupported: 0/127
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/128
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/129
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/130
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/131
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/132
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/133
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/134
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/135
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/136
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/137
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/138
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/139
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/140
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/141
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/142
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/143
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/144
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/145
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/146
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/147
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/148
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/149
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/150
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/151
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/152
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/153
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/154
[ggml_qnn_supports_matmul_op, 452]: [qnn-gpu][MUL_MAT]type src0(1), src1(0) and op(0) are not equal, support/unsupported: 0/155
[ggml_backend_qnn_init_with_device_context, 328]: extend_lib_search_path is nullptr, will use /data/local/tmp/qnn/lib/ as default
[ggml_backend_qnn_init_with_device_context, 333]: device qnn-gpu
[ggml_backend_qnn_init_with_device_context, 334]: extend_lib_search_path /data/local/tmp/qnn/lib/
[ggml_backend_qnn_init_with_device_context, 360]: qnn-gpu backend setenv successfully
[qnn_init, 198]: enter qnn_init

[Gianthard-cyh]

Hi @TerryT9, I encountered the same issue and I'm not sure how to resolve it. Could you share how you solved it?

[chraac]

Hi @TerryT9, I encountered the same issue and I'm not sure how to resolve it. Could you share how you solved it?

Hi @Gianthard-cyh , would you mind to say which model you were using?

[Gianthard-cyh]

Hi @TerryT9, I encountered the same issue and I'm not sure how to resolve it. Could you share how you solved it?

Hi @Gianthard-cyh , would you mind to say which model you were using?

I'm using Llama-3.2-1B-Instruct-f16.gguf. By the way, I run the model successfully after increasing the size limit of MUL_MAT op and changing the precision option of NPU (the execution of Convert op will fail without this) .

However, as stated in previous issues, the NPU backend achives around 1/3 performance of CPU. I think more profiling and optimizing work could be done. I'm happy to help with that.

My device is Oneplus Ace 3 with Snapdragon 8 Gen 2.

--- a/ggml/src/ggml-qnn/graph.cpp
+++ b/ggml/src/ggml-qnn/graph.cpp
@@ -192,8 +192,15 @@ qnn_graph::qnn_graph(const std::string &graph_name, QNNBackend device, std::shar
         graph_vtcm_config.option = QNN_GRAPH_CONFIG_OPTION_CUSTOM;
         graph_vtcm_config.customConfig = &vtcm_config;

+        QnnHtpGraph_CustomConfig_t precision_config;
+        precision_config.option = QNN_HTP_GRAPH_CONFIG_OPTION_PRECISION;
+        precision_config.precision = QNN_PRECISION_FLOAT16;
+        QnnGraph_Config_t graph_precision_config;
+        graph_precision_config.option = QNN_GRAPH_CONFIG_OPTION_CUSTOM;
+        graph_precision_config.customConfig = &precision_config;
+
         const QnnGraph_Config_t *graph_configs[] = {&graph_hvx_config, &graph_dlbc_config, &graph_vtcm_config,
-                                                    &graph_opt_config, nullptr};
+                                                    &graph_opt_config, &graph_precision_config, nullptr};
         error = qnn_interface->qnn_graph_create(qnn_context, graph_name.c_str(), graph_configs, &graph_handle);
     } else {
         error = qnn_interface->qnn_graph_create(qnn_context, graph_name.c_str(), nullptr, &graph_handle);

[chraac]

Hi @TerryT9, I encountered the same issue and I'm not sure how to resolve it. Could you share how you solved it?

Hi @Gianthard-cyh , would you mind to say which model you were using?

I'm using Llama-3.2-1B-Instruct-f16.gguf. By the way, I run the model successfully after increasing the size limit of MUL_MAT op and changing the precision option of NPU (the execution of Convert op will fail without this) .

However, as stated in previous issues, the NPU backend achives around 1/3 performance of CPU. I think more profiling and optimizing work could be done. I'm happy to help with that.

My device is Oneplus Ace 3 with Snapdragon 8 Gen 2.

Nice one, you thing we can force the NPU use the F16 event in F32 mode?

I suggest we create an issue about this and try your configuration.

tbh, while these NPUs claim impressive XX Tflops performance, these are peak numbers limited by various conditions like memory bandwidth. The mul_mat operation is too simple to fully utilize the NPU's processing power.

[Gianthard-cyh]

Hi @TerryT9, I encountered the same issue and I'm not sure how to resolve it. Could you share how you solved it?

Hi @Gianthard-cyh , would you mind to say which model you were using?

I'm using Llama-3.2-1B-Instruct-f16.gguf. By the way, I run the model successfully after increasing the size limit of MUL_MAT op and changing the precision option of NPU (the execution of Convert op will fail without this) .
However, as stated in previous issues, the NPU backend achives around 1/3 performance of CPU. I think more profiling and optimizing work could be done. I'm happy to help with that.
My device is Oneplus Ace 3 with Snapdragon 8 Gen 2.

nice one, you thing we can force the NPU use the F16 event in F32 mode?

thought we can try!

I'm not sure what this option does. It just works after adding this option; otherwise, a strange error occurs when executing the Convert node in the graph. I’ve also tried calling the op validation API before adding the node to the graph — the op configuration passed the validation, but the execution still failed.

[chraac]

I'm not sure what this option does. It just works after adding this option; otherwise, a strange error occurs when executing the Convert node in the graph. I’ve also tried calling the op validation API before adding the node to the graph — the op configuration passed the validation, but the execution still failed.

create an issue for it, we can take there #34