ggml_new_tensor_impl: not enough space in the context's memory pool (needed 7082923680, available 7082732800)

[imSrbh]

@su77ungr
I am having 32Cores and 64GB RAM.

1. I am getting ggml_new_tensor_impl: not enough space in the context's memory pool (needed 7082923680, available 7082732800)

2. How we can restrict the token_length? and limit its domain to the ingested document file?

> Question:
who is saying that "save democracy"

 

> Answer:
The speaker is calling for the Senate to pass the Freedom to Vote Act, the John Lewis Voting Rights Act, and the Disclose Act to ensure that Americans have the right to vote and to know who is funding their elections.

 

> Time Taken: 39.02538466453552

 

Enter a query: what is the date today?

 

llama_print_timings:        load time =   227.71 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time =   334.69 ms /     7 tokens (   47.81 ms per token)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings:       total time =   337.46 ms
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
gpt_tokenize: unknown token '�'
ggml_new_tensor_impl: not enough space in the context's memory pool (needed 7082923680, available 7082732800)

[su77ungr]

1.Q
This seems to be a llamacpp related issue see
Make sure you are running the latest llamacpp python module. I'm on it.

2.Q
You can play around with the class of LlamaCppEmbeddings

class LlamaCpp(LLM):
    """Wrapper around the llama.cpp model.

    To use, you should have the llama-cpp-python library installed, and provide the
    path to the Llama model as a named parameter to the constructor.
    Check out: https://github.com/abetlen/llama-cpp-python

    Example:
        .. code-block:: python

            from langchain.llms import LlamaCppEmbeddings
            llm = LlamaCppEmbeddings(model_path="/path/to/llama/model")
    """

    client: Any  #: :meta private:
    model_path: str
    """The path to the Llama model file."""

    lora_base: Optional[str] = None
    """The path to the Llama LoRA base model."""

    lora_path: Optional[str] = None
    """The path to the Llama LoRA. If None, no LoRa is loaded."""

    n_ctx: int = Field(512, alias="n_ctx")
    """Token context window."""

    n_parts: int = Field(-1, alias="n_parts")
    """Number of parts to split the model into.
    If -1, the number of parts is automatically determined."""

    seed: int = Field(-1, alias="seed")
    """Seed. If -1, a random seed is used."""

    f16_kv: bool = Field(True, alias="f16_kv")
    """Use half-precision for key/value cache."""

    logits_all: bool = Field(False, alias="logits_all")
    """Return logits for all tokens, not just the last token."""

    vocab_only: bool = Field(False, alias="vocab_only")
    """Only load the vocabulary, no weights."""

    use_mlock: bool = Field(False, alias="use_mlock")
    """Force system to keep model in RAM."""

    n_threads: Optional[int] = Field(None, alias="n_threads")
    """Number of threads to use.
    If None, the number of threads is automatically determined."""

    n_batch: Optional[int] = Field(8, alias="n_batch")
    """Number of tokens to process in parallel.
    Should be a number between 1 and n_ctx."""

    suffix: Optional[str] = Field(None)
    """A suffix to append to the generated text. If None, no suffix is appended."""

    max_tokens: Optional[int] = 256
    """The maximum number of tokens to generate."""
    
    ```

You might try to handle it with inside startLLM.py

change it to :

 llama = LlamaCppEmbeddings(model_path='./models/ggml-model-q4_0.bin', max_tokens=128)

[imSrbh]

@su77ungr I am still getting this error..

Enter a query: what is vickey's email?

llama_print_timings:        load time =   200.62 ms
llama_print_timings:      sample time =     0.00 ms /     1 runs   (    0.00 ms per run)
llama_print_timings: prompt eval time =   429.41 ms /     8 tokens (   53.68 ms per token)
llama_print_timings:        eval time =   116.50 ms /     1 runs   (  116.50 ms per run)
llama_print_timings:       total time =   551.65 ms
ggml_new_tensor_impl: not enough space in the context's memory pool (needed 6426660032, available 6418726608)
Segmentation fault (core dumped)

I am using latest changes only as you suggested about Llama module version.

[su77ungr]

That's your hardware bottlenecking. Try to ingest smaller files or upgrade.

[imSrbh]

I am using sample.csv provided in the source_documents. It's smaller file only.

Its 64GB RAM and 32Cores cpu.
Got the same in RTX3050 16Cores and 16GB RAM.

[su77ungr]

Try building from docker. This is the first error I could not recreate. I'm running on 16GiB without a problem though.

I'm searching for a fix

[hippalectryon-0]

@imSrbh Does it still happen with the new release ?

[imSrbh]

@hippalectryon-0

No, It's not with the new release.
The only issue I am facing is GUI:

there is no attribute in startLLM named initialize_qa_system but still it's been used in gui.

[hippalectryon-0]

Your new error is fixed in #58 , it's awaiting merge.

If your original issue is fixed, you can close the issue :)