convert-hf-to-gguf.py Qwen-72B-Chat model get Killed result

[dfengpo]

I use python convert-hf-to-gguf.py /Qwen-72B-Chat.
And I am getting the same error:
blk.33.ffn_down.weight, n_dims = 2, torch.bfloat16 --> float16 blk.33.ffn_up.weight, n_dims = 2, torch.bfloat16 --> float16 blk.33.ffn_gate.weight, n_dims = 2, torch.bfloat16 --> float16 blk.34.attn_qkv.bias, n_dims = 1, torch.bfloat16 --> float32 blk.34.attn_qkv.weight, n_dims = 2, torch.bfloat16 --> float16 blk.34.attn_output.weight, n_dims = 2, torch.bfloat16 --> float16 blk.34.attn_norm.weight, n_dims = 1, torch.bfloat16 --> float32 blk.34.ffn_norm.weight, n_dims = 1, torch.bfloat16 --> float32 blk.34.ffn_up.weight, n_dims = 2, torch.bfloat16 --> float16 blk.34.ffn_down.weight, n_dims = 2, torch.bfloat16 --> float16 blk.34.ffn_gate.weight, n_dims = 2, torch.bfloat16 --> float16 blk.35.attn_qkv.bias, n_dims = 1, torch.bfloat16 --> float32 blk.35.attn_qkv.weight, n_dims = 2, torch.bfloat16 --> float16 blk.35.attn_output.weight, n_dims = 2, torch.bfloat16 --> float16 blk.35.attn_norm.weight, n_dims = 1, torch.bfloat16 --> float32 blk.35.ffn_norm.weight, n_dims = 1, torch.bfloat16 --> float32 Killed

What does this mean “Killed”?
@ggerganov @slaren @prusnak

[prusnak]

I assume you have run out of memory. How much RAM do you have?

[dfengpo]

I assume you have run out of memory. How much RAM do you have?

My RAM is 64Gb，cpu32

[ngxson]

The process is likely to be killed because of low memory: https://stackoverflow.com/questions/726690/what-killed-my-process-and-why

[Galunid]

It seems like a bug, and the offending line is
https://github.com/ggerganov/llama.cpp/blob/a1d6df129bcd3d42cda38c09217d8d4ec4ea3bdd/convert-hf-to-gguf.py#L999

Which makes the script load the whole (80GB+) model into memory, instead of using mmap from torch

@lmxin123 Could you try the script with the changes below?

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 7a0a8c3d..8cef8429 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -996,9 +996,8 @@ class QwenModel(Model):
 
     def write_tensors(self):
         block_count = self.hparams["num_hidden_layers"]
-        model_kv = dict(self.get_tensors())
         tensor_map = gguf.get_tensor_name_map(self.model_arch, block_count)
-        for name, data_torch in model_kv.items():
+        for name, data_torch in self.get_tensors():
             # we don't need these
             if name.endswith(".rotary_emb.inv_freq"):
                 continue

[dfengpo]

It seems like a bug, and the offending line is

https://github.com/ggerganov/llama.cpp/blob/a1d6df129bcd3d42cda38c09217d8d4ec4ea3bdd/convert-hf-to-gguf.py#L999

Which makes the script load the whole (80GB+) model into memory, instead of using mmap from torch

@lmxin123 Could you try the script with the changes below?

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 7a0a8c3d..8cef8429 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -996,9 +996,8 @@ class QwenModel(Model):
 
     def write_tensors(self):
         block_count = self.hparams["num_hidden_layers"]
-        model_kv = dict(self.get_tensors())
         tensor_map = gguf.get_tensor_name_map(self.model_arch, block_count)
-        for name, data_torch in model_kv.items():
+        for name, data_torch in self.get_tensors():
             # we don't need these
             if name.endswith(".rotary_emb.inv_freq"):
                 continue

Thank you for your response, but unfortunately, I don't know Python, and I'm unable to test your modifications. I look forward to an official update of the version.

[timopb]

It seems like a bug, and the offending line is

https://github.com/ggerganov/llama.cpp/blob/a1d6df129bcd3d42cda38c09217d8d4ec4ea3bdd/convert-hf-to-gguf.py#L999

Which makes the script load the whole (80GB+) model into memory, instead of using mmap from torch

@lmxin123 Could you try the script with the changes below?

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 7a0a8c3d..8cef8429 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -996,9 +996,8 @@ class QwenModel(Model):
 
     def write_tensors(self):
         block_count = self.hparams["num_hidden_layers"]
-        model_kv = dict(self.get_tensors())
         tensor_map = gguf.get_tensor_name_map(self.model_arch, block_count)
-        for name, data_torch in model_kv.items():
+        for name, data_torch in self.get_tensors():
             # we don't need these
             if name.endswith(".rotary_emb.inv_freq"):
                 continue

Having the same issue with converting falcon-40b on a machine with 24GB RAM. Process gets killed most likely due to lack of memory. I applied the patch from your response but it didn't help unfortunately.

[Galunid]

@lmxin123 Could you try with this script? https://gist.github.com/Galunid/c169dd4078c9cb11e8d8a4a8888eab2b
Just copy the contents into convert-hf-to-gguf.py and run it like you normally would.

[Galunid]

@timopb Falcon is a separate issue and the above is not applicable.

[arch-btw]

I'm having the same problem even with the new script by @Galunid

It's not just loading the original model into ram, it's also writing the new model to ram first, instead of disk.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.