Convert-hf-to-gguf fails with command-r-plus

[candre23]

Trying to quantize the just-released Command R+ model. I know command R support was added a while back, but there appears to be something different about this new, bigger model that is causing issues. With a fresh clone of LCPP from a few minutes ago, this is the failure I get when trying to convert.

Traceback (most recent call last):
  File "G:\lcpp2\convert-hf-to-gguf.py", line 2443, in <module>
    main()
  File "G:\lcpp2\convert-hf-to-gguf.py", line 2424, in main
    model_instance = model_class(dir_model, ftype_map[args.outtype], fname_out, args.bigendian)
  File "G:\lcpp2\convert-hf-to-gguf.py", line 2347, in __init__
    self.hparams["max_position_embeddings"] = self.hparams["model_max_length"]
KeyError: 'model_max_length'

https://huggingface.co/CohereForAI/c4ai-command-r-plus

[bartowski1182]

I fixed this by including that line in config.JSON, however it still fails after doing that for a different reason:

Can not map tensor 'model.layers.0.self_attn.k_norm.weight'

[candre23]

Yes, I just found the same thing after it was suggested to add the max length. New output in its entirety:

gguf: This GGUF file is for Little Endian only
Set model parameters
gguf: context length = 131072
gguf: embedding length = 12288
gguf: feed forward length = 33792
gguf: head count = 96
gguf: key-value head count = 8
gguf: rope theta = 75000000.0
gguf: layer norm epsilon = 1e-05
gguf: file type = 1
Set model tokenizer
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
gguf: Adding 253333 merge(s).
gguf: Setting special token type bos to 5
gguf: Setting special token type eos to 255001
gguf: Setting special token type pad to 0
gguf: Setting add_bos_token to True
gguf: Setting add_eos_token to False
Exporting model to 'e:\Command-R-Plus_fp16.gguf'
gguf: loading model part 'model-00001-of-00044.safetensors'
token_embd.weight, n_dims = 2, torch.float16 --> float16
gguf: loading model part 'model-00002-of-00044.safetensors'
blk.0.attn_norm.weight, n_dims = 1, torch.float16 --> float32
blk.0.ffn_down.weight, n_dims = 2, torch.float16 --> float16
blk.0.ffn_gate.weight, n_dims = 2, torch.float16 --> float16
blk.0.ffn_up.weight, n_dims = 2, torch.float16 --> float16
Can not map tensor 'model.layers.0.self_attn.k_norm.weight'

[RefractAI]

#6491

This fixes the tensor map error.

[candre23]

Yeah, but it still fails per @bartowski1182 's attempts. Everybody's chomping at the bit for this beast.

[ehartford]

For the mean time, if you are on mac there is https://huggingface.co/mlx-community/c4ai-command-r-plus-4bit

[slaren]

The map error is probably caused by an outdated gguf-py. The recommendation is to install the local version.