Eval bug: Qwen3 30B A3B Q4_0 failed to run

[riteme]

Name and Version

version: 5215 (5f5e39e)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CPU

Hardware

Intel(R) Core(TM) i9-14900K

Models

Qwen3 30B A3B q4_0

Problem description & steps to reproduce

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-cli llama-quantize --config Release -j32
python convert_hf_to_gguf.py /data/models/qwen3_30b_a3b --outtype f16 --outfile /data/models/qwen3_30b_a3b.gguf
./build/bin/llama-quantize /data/models/qwen3_30b_a3b.gguf /data/models/qwen3_30b_a3b_q4_0.gguf q4_0 16
./build/bin/llama-cli -m /data/models/qwen3_30b_a3b_q4_0.gguf --temp 0.6 --top-p 0.95 --top
-k 20 --min-p 0

First Bad Commit

No response

Relevant log output

build: 5215 (5f5e39e1) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 36 key-value pairs and 579 tensors from /data/models/qwen3_30b_a3b_q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3_30B_A3B
llama_model_loader: - kv   3:                         general.size_label str              = 128x1.8B
llama_model_loader: - kv   4:                            general.license str              = apache-2.0
llama_model_loader: - kv   5:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
llama_model_loader: - kv   7:                  general.base_model.0.name str              = Qwen3 30B A3B Base
llama_model_loader: - kv   8:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv  10:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv  11:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  12:                    qwen3moe.context_length u32              = 40960
llama_model_loader: - kv  13:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  14:               qwen3moe.feed_forward_length u32              = 6144
llama_model_loader: - kv  15:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  16:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  17:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  19:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  20:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  21:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  22:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  23:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  32:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q4_0:  337 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 16.11 GiB (4.53 BPW) 
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: n_swa_pattern    = 1
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 6144
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 30.53 B
print_info: general.name     = Qwen3_30B_A3B
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:  CPU_AARCH64 model buffer size = 16038.00 MiB
load_tensors:   CPU_Mapped model buffer size = 16389.15 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1
init:        CPU KV buffer size =   384.00 MiB
llama_context: KV self size  =  384.00 MiB, K (f16):  192.00 MiB, V (f16):  192.00 MiB
llama_context:        CPU compute buffer size =   300.75 MiB
llama_context: graph nodes  = 3126
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
/data/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp:6186: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
[New LWP 842910]
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/liblber.so.2
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libbrotlidec.so.1
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libbrotlicommon.so.1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000728b186e37e3 in __GI___wait4 (pid=842941, stat_loc=0x7fff84a79e14, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000728b186e37e3 in __GI___wait4 (pid=842941, stat_loc=0x7fff84a79e14, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000728b18c5ca0b in ggml_abort () from /data/llama.cpp/build/bin/libggml-base.so
#2  0x0000728b1851b295 in ggml::cpu::aarch64::tensor_traits<block_q4_0, 8l, 8l, (ggml_type)8>::compute_forward(ggml_compute_params*, ggml_tensor*) () from /data/llama.cpp/build/bin/libggml-cpu.so
#3  0x0000728b18522986 in ggml_cpu_extra_compute_forward () from /data/llama.cpp/build/bin/libggml-cpu.so
#4  0x0000728b185062ea in ggml_graph_compute_thread.isra () from /data/llama.cpp/build/bin/libggml-cpu.so
#5  0x0000728b1850b85e in ggml_graph_compute () from /data/llama.cpp/build/bin/libggml-cpu.so
#6  0x0000728b1850bc06 in ggml_backend_cpu_graph_compute(ggml_backend*, ggml_cgraph*) () from /data/llama.cpp/build/bin/libggml-cpu.so
#7  0x0000728b18c72713 in ggml_backend_sched_graph_compute_async () from /data/llama.cpp/build/bin/libggml-base.so
#8  0x0000728b18da0ba1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /data/llama.cpp/build/bin/libllama.so
#9  0x0000728b18da2f58 in llama_context::decode(llama_batch&) () from /data/llama.cpp/build/bin/libllama.so
#10 0x0000728b18da435f in llama_decode () from /data/llama.cpp/build/bin/libllama.so
#11 0x000058cf0d49161a in common_init_from_params(common_params&) ()
#12 0x000058cf0d3a32aa in main ()
[Inferior 1 (process 842909) detached]

[ymcki]

I am on b5224 but I can't convert using convert_hf_to_gguf.py.

python3 convert_hf_to_gguf.py ~/Qwen3-30B-A3B-FP8/ --outtype f16 --outfile x.gguf
INFO:hf-to-gguf:Loading model: Qwen3-30B-A3B-FP8
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00007.safetensors'
INFO:hf-to-gguf:token_embd.weight,           torch.float32 --> F16, shape = {2048, 151936}
INFO:hf-to-gguf:blk.0.attn_norm.weight,      torch.float32 --> F32, shape = {2048}
Traceback (most recent call last):
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 5904, in <module>
    main()
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 5898, in main
    model_instance.write()
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 400, in write
    self.prepare_tensors()
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 2663, in prepare_tensors
    super().prepare_tensors()
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 276, in prepare_tensors
    for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/llama.cpp.new/convert_hf_to_gguf.py", line 2646, in modify_tensors
    datas.append(self._experts[bid][ename])
                 ~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'model.layers.0.mlp.experts.5.down_proj.weight'

[CISC]

I am on b5224 but I can't convert using convert_hf_to_gguf.py.

python3 convert_hf_to_gguf.py ~/Qwen3-30B-A3B-FP8/ --outtype f16 --outfile x.gguf

FP8 is not supported, get the normal BF16 model.

[CISC]

@riteme Is this still an issue, or can this be closed?

[riteme]

Yes, it persists in b5703:

build: 5703 (10bb545c) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 36 key-value pairs and 579 tensors from /data/models/qwen3_30b_a3b_q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3_30B_A3B
llama_model_loader: - kv   3:                         general.size_label str              = 128x1.8B
llama_model_loader: - kv   4:                            general.license str              = apache-2.0
llama_model_loader: - kv   5:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
llama_model_loader: - kv   7:                  general.base_model.0.name str              = Qwen3 30B A3B Base
llama_model_loader: - kv   8:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-30B...
llama_model_loader: - kv  10:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv  11:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  12:                    qwen3moe.context_length u32              = 40960
llama_model_loader: - kv  13:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  14:               qwen3moe.feed_forward_length u32              = 6144
llama_model_loader: - kv  15:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  16:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  17:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  19:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  20:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  21:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  22:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  23:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  32:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q4_0:  337 tensors
llama_model_loader: - type q6_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 16.11 GiB (4.53 BPW) 
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 6144
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 30B.A3B
print_info: model params     = 30.53 B
print_info: general.name     = Qwen3_30B_A3B
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors:   CPU_REPACK model buffer size = 16038.00 MiB
load_tensors:   CPU_Mapped model buffer size = 16389.15 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache_unified:        CPU KV buffer size =   384.00 MiB
llama_kv_cache_unified: size =  384.00 MiB (  4096 cells,  48 layers,  1 seqs), K (f16):  192.00 MiB, V (f16):  192.00 MiB
llama_context:        CPU compute buffer size =   300.75 MiB
llama_context: graph nodes  = 3222
llama_context: graph splits = 1
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
/root/llama.cpp/ggml/src/ggml-cpu/repack.cpp:1308: GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) + n_as * ne12 * sizeof(mmid_row_mapping))) failed
[New LWP 776692]
[New LWP 776691]
[New LWP 776690]
[New LWP 776689]
[New LWP 776688]
[New LWP 776687]
[New LWP 776686]
[New LWP 776685]
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
warning: process 776684 is already traced by process 776693
ptrace: Operation not permitted.
No stack.
The program is not being run.
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/liblber.so.2
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libbrotlidec.so.1
warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libbrotlicommon.so.1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000073b4412b07e3 in __GI___wait4 (pid=776693, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x000073b4412b07e3 in __GI___wait4 (pid=776693, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30      in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x000073b44182ad23 in ggml_print_backtrace () from /root/llama.cpp/build/bin/libggml-base.so
#2  0x000073b44182ae3f in ggml_abort () from /root/llama.cpp/build/bin/libggml-base.so
#3  0x000073b4410d2595 in ggml::cpu::repack::tensor_traits<block_q4_0, 8l, 8l, (ggml_type)8>::compute_forward(ggml_compute_params*, ggml_tensor*) () from /root/llama.cpp/build/bin/libggml-cpu.so
#4  0x000073b4410dab36 in ggml_cpu_extra_compute_forward () from /root/llama.cpp/build/bin/libggml-cpu.so
#5  0x000073b4410c15aa in ggml_graph_compute_thread.isra () from /root/llama.cpp/build/bin/libggml-cpu.so
#6  0x000073b44081c977 in GOMP_parallel () from /lib/x86_64-linux-gnu/libgomp.so.1
#7  0x000073b4410c6a13 in ggml_graph_compute () from /root/llama.cpp/build/bin/libggml-cpu.so
#8  0x000073b4410c6e86 in ggml_backend_cpu_graph_compute(ggml_backend*, ggml_cgraph*) () from /root/llama.cpp/build/bin/libggml-cpu.so
#9  0x000073b441840c93 in ggml_backend_sched_graph_compute_async () from /root/llama.cpp/build/bin/libggml-base.so
#10 0x000073b441980ca1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /root/llama.cpp/build/bin/libllama.so
#11 0x000073b4419812a3 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_state_i*, ggml_status&) () from /root/llama.cpp/build/bin/libllama.so
#12 0x000073b441984d08 in llama_context::decode(llama_batch const&) () from /root/llama.cpp/build/bin/libllama.so
#13 0x000073b441985fcf in llama_decode () from /root/llama.cpp/build/bin/libllama.so
#14 0x0000638c9151fb2a in common_init_from_params(common_params&) ()
#15 0x0000638c9142dcc5 in main ()
[Inferior 1 (process 776684) detached]
fish: Job 1, './build/bin/llama-cli -m /data/…' terminated by signal SIGABRT (Abort)

[CISC]

@xctan Can you take a look?

[ggerganov]

I tried to take a look, but I seem to have forgotten how to enable repacking on Mac. Even when I build without Metal, I still get:

make -j && ./bin/llama-cli -m ../models/qwen3-30b-a3b/ggml-model-q4_0.gguf --no-mmap

...

0.00.116.304 D load_tensors: tensor 'token_embd.weight' (q4_0) (and 578 others) cannot be used with preferred buffer type CPU_REPACK, using CPU instead
0.00.116.386 I load_tensors:          CPU model buffer size = 16420.65 MiB

...

0.02.873.551 I system_info: n_threads = 16 (n_threads_batch = 16) / 24 | CPU : ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | REPACK = 1 | 

Was there any extra steps needed?

[ggerganov]

I got the repacking to work, but I don't reproduce the error reported here.

[riteme]

After investigation, the issue arise from a mismatch between:

llama.cpp/ggml/src/ggml-cpu/repack.cpp

Lines 1168 to 1172 in 9eaa51e

	case GGML_OP_MUL_MAT_ID:
	size = ggml_row_size(PARAM_TYPE, ggml_nelements(op->src[1]));
	size = GGML_PAD(size, sizeof(int64_t)); // + padding for next bloc.
	size += sizeof(int64_t) * (1+op->src[0]->ne[2]) * op->src[1]->ne[2];
	return true;

which expands to ggml_row_size(ne10*ne11*ne12*ne13) + ne12*8 + ne02*ne12*8, and the assertion:

llama.cpp/ggml/src/ggml-cpu/repack.cpp

Lines 1308 to 1309 in 9eaa51e

	GGML_ASSERT(params->wsize >= (GGML_PAD(nbw3, sizeof(int64_t)) + n_as * sizeof(int64_t) +
	n_as * ne12 * sizeof(mmid_row_mapping)));

which expands to ggml_row_size(ne10)*ne11*ne12 + ne02*8 + ne02*ne12*8. The second term differs (ne12*8 vs ne02*8).

Rewriting L1171 to size += sizeof(int64_t) * (1+op->src[1]->ne[2]) * op->src[0]->ne[2]; resolves this assertion and the model runs correctly, though I'm not certain whether this is the precise fix.

[ggerganov]

@riteme Thanks for tracking this down. Could you confirm that #14292 fixes the issue?

[riteme]

@ggerganov Yes, I confirm #14292 fixes the issue.