GGML_ASSERT: llama.cpp:13518: (qs.n_attention_wv == 0 || qs.n_attention_wv == (int)model.hparams.n_layer) && "n_attention_wv is unexpected"

[schmorp]

I got this when trying to make a Q8_0 quant of https://huggingface.co/Noodlz/DolphinLake-7B

When using main, it also fails:

llama_model_load: error loading model: check_tensor_dims: tensor 'blk.1.ffn_down.weight' not found

Assuming that the model itself is broken (without claiming that it is), maybe convert.py could detect missing tensors or something like that?

If it's a broken model and everything acts as it should, apologies.

I have the same upon ./quantize ./models/PULI-LlumiX-32K-instruct/ggml-model-f32.gguf ./models/PULI-LlumiX-32K-instruct/PULI-LlumiX-32K-instruct-Q2_K.gguf Q2_K. Or any other Q parameters.

Seems to work with other models, at least f16 gguf-s.

[compilade]

@schmorp Are you sure all of the parts of the model, from model-00001-of-00008.safetensors to model-00008-of-00008.safetensors have been downloaded and are present in your model directory?

This sounds very similar to #7215

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.