no support for loading 70B models via llamacpp

[yairVanti]

System Info

LangChain 0.0.242

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

tried to load https://huggingface.co/TheBloke/StableBeluga2-70B-GGML model to work with lanchain's Llamacpp :

llm = LlamaCpp(model_path="./stablebeluga2-70b.ggmlv3.q4_0.bin", n_gpu_layers=n_gpu_layers, n_batch=n_batch, n_ctx=8192, input={"temperature": 0.01},n_threads=8) llm_chain = LLMChain(llm=llm, prompt=prompt)

i see that there is no support in passing n_gqa=8 parameter that according to https://github.com/abetlen/llama-cpp-python should be used for 70B models
the error i get is :
error loading model: llama.cpp: tensor 'layers.0.attention.wk.weight' has wrong shape; expected 8192 x 8192, got 8192 x 1024

Expected behavior

model should be loaded successfuly

[william-incode]

+1 I'm having the same issue, the n_gqa arg is not working in LlamaCpp

[ankuratudemy]

embeddings = LlamaCppEmbeddings(model_path="..../llama-2-70b.ggmlv3.q4_0.bin", n_gqa=8). n_gqa is not suported in LlamaCppEmbeddings yet.

Support for n_gqa added in : abetlen/llama-cpp-python#539

Workaround for now :

Edit llamacpp.py:

Add:

n_gqa: Optional[int] = Field(None, alias="n_gqa")
    """grouped-query attention"""

&

model_param_names = [
            "n_ctx",
            "n_parts",
            "seed",
            "f16_kv",
            "logits_all",
            "vocab_only",
            "use_mlock",
            "n_threads",
            "n_batch",
            **"n_gqa"**
        ]

Then using LlamaCppEmbeddings(model_path="..../llama-2-70b.ggmlv3.q4_0.bin", n_gqa=8) works,

Output:

llama_model_load_internal: warning: assuming 70B model based on GQA == 8
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 8192
llama_model_load_internal: n_mult     = 4096
llama_model_load_internal: n_head     = 64
llama_model_load_internal: n_head_kv  = 8
llama_model_load_internal: n_layer    = 80
llama_model_load_internal: n_rot      = 128
**llama_model_load_internal: n_gqa      = 8**
llama_model_load_internal: rnorm_eps  = 1.0e-06
llama_model_load_internal: n_ff       = 28672
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 70B
llama_model_load_internal: ggml ctx size =    0.21 MB
llama_model_load_internal: mem required  = 37635.96 MB (+  320.00 MB per state)
llama_new_context_with_model: kv self size  =  320.00 MB
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | 

[dosubot]

Hi, @yairVanti! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding, you are experiencing an error when trying to load a 70B model using llamacpp. It seems that llamacpp does not support the parameter called n_gqa, which is required for 70B models. Another user, @william-incode, has also reported the same issue. However, @ankuratudemy has suggested a workaround by editing llamacpp.py to add support for the required parameter n_gqa.

Before we close this issue, we wanted to check if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!