NameError: Could not load Llama model from path: D:\privateGPT\ggml-model-q4_0.bin

[michael7908]

I checked this issue with GPT-4 and this is what I got:

The error message is indicating that the Llama model you're trying to use is in an old format that is no longer supported. The error message suggests to visit a URL for more information: ggml-org/llama.cpp#1305.

As of my knowledge cutoff in September 2021, I can't provide direct insight into the specific contents of that pull request or the subsequent changes in the Llama library. You should visit the URL provided in the error message for the most accurate and up-to-date information.

However, based on the error message, it seems like you need to convert your Llama model to a new format that is supported by the current version of the Llama library. You should look for documentation or tools provided by the Llama library that can help you perform this conversion.

If the Llama model (ggml-model-q4_0.bin) was provided to you or downloaded from a third-party source, you might also want to check if there's an updated version of the model available in the new format.

Could you please help me out on this? Thank you in advance.

[michael7908]

The whole error message:

PS D:\privateGPT> python ingest.py
Loading documents from source_documents
Loaded 2 documents from source_documents
Split into 91 chunks of text (max. 500 tokens each)
llama.cpp: loading model from D:\privateGPT\ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1024
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
error loading model: this format is no longer supported (see ggml-org/llama.cpp#1305)
llama_init_from_file: failed to load model
Traceback (most recent call last):
File "C:\Python311\Lib\site-packages\langchain\embeddings\llamacpp.py", line 78, in validate_environment
values["client"] = Llama(
^^^^^^
File "C:\Python311\Lib\site-packages\llama_cpp\llama.py", line 161, in init
assert self.ctx is not None
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\privateGPT\ingest.py", line 62, in
main()
File "D:\privateGPT\ingest.py", line 53, in main
llama = LlamaCppEmbeddings(model_path=llama_embeddings_model, n_ctx=model_n_ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pydantic\main.py", line 339, in pydantic.main.BaseModel.init
File "pydantic\main.py", line 1102, in pydantic.main.validate_model
File "C:\Python311\Lib\site-packages\langchain\embeddings\llamacpp.py", line 98, in validate_environment
raise NameError(f"Could not load Llama model from path: {model_path}")
NameError: Could not load Llama model from path: D:\privateGPT\ggml-model-q4_0.bin

[Mostajerane]

I also have the same issue, can anyone help?

[Mostajerane]

@michael7908 create a new environment, install the requirements, this will solve the issue.

[michael7908]

Hi Thanks, do you mean a virtual environment? thanks

…

On Sun, May 14, 2023 at 9:06 PM Mostajerane ***@***.***> wrote: @michael7908 <https://github.com/michael7908> create a new environment, install the requirements, this will solve the issue. — Reply to this email directly, view it on GitHub <#113 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3L6MEAAEXA3UWHWHV7TKDXGDKF3ANCNFSM6AAAAAAYBBI3TY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Mostajerane]

Yes

[pboethig2]

use conda an conda create

[maozdemir]

Creating a new environment is not a solution. See ggml-org/llama.cpp#1305

[YangZeyu95]

pip install llama-cpp-python==0.1.48 resolved my issue

[ChatTeach]

ya...its very useful.

i solved my issue.

[inventivejon]

It also solved it for me

[augusto-rehfeldt]

EDIT: fixed by installing llama-cpp-python > 0.1.53! Thanks!

---

Hello, it didn't solve the issue for me.

My python version is 3.11.0.

I'm using Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin inside "models", which is a GGML v3 model, and llama-cpp-python version 0.1.52.

Error log in powershell:

PS C:\llm\privateGPT> python .\privateGPT.py
Using embedded DuckDB with persistence: data will be stored in: db
llama.cpp: loading model from models/Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model
Traceback (most recent call last):
  File "C:\llm\privateGPT\privateGPT.py", line 75, in <module>
    main()
  File "C:\llm\privateGPT\privateGPT.py", line 33, in main
    llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic\main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
__root__
  Could not load Llama model from path: models/Wizard-Vicuna-7B-Uncensored.ggmlv3.q4_0.bin. Received error  (type=value_error)

I've already tried reinstalling llama-cpp-python with different versions.

Thanks for your help.

[UnathiNeo]

I was able to solve this issue by using pip install llama-cpp-python==0.1.53

Using embedded DuckDB with persistence: data will be stored in: db
llama.cpp: loading model from Models/koala-7B.ggmlv3.q4_0.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model
Traceback (most recent call last):
File "C:\Users\Desktop\Desktop\Demo\privateGPT\privateGPT.py", line 75, in
main()
File "C:\Users\Desktop\Desktop\Demo\privateGPT\privateGPT.py", line 33, in main
llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
root
Could not load Llama model from path: Models/koala-7B.ggmlv3.q4_0.bin. Received error (type=value_error)
PS C:\Users\Desktop\Desktop\Demo\privateGPT> pip install llama-cpp-python==0.1.53
Collecting llama-cpp-python==0.1.53
Downloading llama_cpp_python-0.1.53.tar.gz (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 172.4 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in c:\users\desktop\appdata\local\programs\python\python310\lib\site-packages (from llama-cpp-python==0.1.53) (4.6.0)
Building wheels for collected packages: llama-cpp-python
Building wheel for llama-cpp-python (pyproject.toml) ... done
Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.53-cp310-cp310-win_amd64.whl size=255379 sha256=f12fcbb823810374109b5c1e690570899cb72c73fd03dae1d95fa1b990878dd7
Stored in directory: c:\users\desktop\appdata\local\pip\cache\wheels\a8\92\29\90f6353e5d588d26c7f7d9656951f24b3d0e8eba24f6d6fbce
Successfully built llama-cpp-python
Installing collected packages: llama-cpp-python
Attempting uninstall: llama-cpp-python
Found existing installation: llama-cpp-python 0.1.52
Uninstalling llama-cpp-python-0.1.52:
Successfully uninstalled llama-cpp-python-0.1.52
Successfully installed llama-cpp-python-0.1.53
PS C:\Users\Desktop\Desktop\Demo\privateGPT> python privateGPT.py
Using embedded DuckDB with persistence: data will be stored in: db
llama.cpp: loading model from Models/koala-7B.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1000
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 0.07 MB
llama_model_load_internal: mem required = 5407.71 MB (+ 1026.00 MB per state)
.
llama_init_from_file: kv self size = 500.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

[Rakeshcool]

yep thanks it worked

[aiornothing]

great, <pip install llama-cpp-python==0.1.53> worked for me too!!!

[GirishKiranH]

@augusto-rehfeldt am getting similar issue , did it worked for you ? am not able to load ggml-nous-gpt4-vicuna-13b or similar llama models on my M1 Macbook, can anyone help here ?
Am getting below error, i tried llama-cpp-python with both 0.1.53 and 0.1.48 , but no luck

llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp
__root__