Error: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

[irthomasthomas]

Hey, I'm not sure if this is a bug, or I just need change a config somewhere, but since the update, I get this error when I try to run models split across two cuda devices:

ValueError: Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES=1

Is that related to this? ggml-org/llama.cpp#5240

Here is the command I ran:
python3 -m llama_cpp.server --model /Models/deepseek-coder-33b-instruct.Q4_K_M.gguf --n_gpu_layers 56 --tensor_split 64 36 --offload_kqv false --n_ctx 8000 --n_batch 56 --chat_format chatml

[hamza233]

Getting same error...

[bani6809]

same error here. I have 4 x 3090 and it won't let me use them.

[abetlen]

Hey sorry for the late reply, this shouldn't be related to that issue as I'd already migrated to using the llama_max_devices function. Can you try running python3 -c "import llama_cpp; print(llama_cpp.llama_max_devices())"?

[bani6809]

admin@llm:~/models$ python3 -c "import llama_cpp; print(llama_cpp.llama_max_devices())"
1
admin@llm:~$ python3 -c "import torch; print(torch.cuda.device_count())"
5

my current setup has 5 x 3090 but llama_cpp won't use them 😭

[abetlen]

Looking into it a bit these are certainly caused by the library not building with any gpu support,

size_t llama_max_devices(void) {
#if defined(GGML_USE_METAL)
    return 1;
#elif defined(GGML_USE_CUBLAS)
    return GGML_CUDA_MAX_DEVICES;
#elif defined(GGML_USE_SYCL)
    return GGML_SYCL_MAX_DEVICES;
#elif defined(GGML_USE_VULKAN)
    return GGML_VK_MAX_DEVICES;
#else
    return 1;
#endif
}

llama_max_devices just returns the value of GGML_CUDA_MAX_DEVICES which should be 16 if you successfully compiled with CUBLAS. I would reinstall with the build flags from the README and if that's still not working post your build log CMAKE_ARGS="..." pip install llama-cpp-python --verbose

[thiner]

in llama_cpp.py, below code fails everytime. it's still referencing LLAMA_MAX_DEVICES, rather than function llama_max_devices(). Is this the root cause?

 if self.tensor_split is not None:
            if len(self.tensor_split) > llama_cpp.LLAMA_MAX_DEVICES:
                raise ValueError(
                    f"Attempt to split tensors that exceed maximum supported devices. Current LLAMA_MAX_DEVICES={llama_cpp.LLAMA_MAX_DEVICES}"
                )

[cebtenzzre]

in llama_cpp.py, below code fails everytime. it's still referencing LLAMA_MAX_DEVICES, rather than function llama_max_devices(). Is this the root cause?

No, LLAMA_MAX_DEVICES comes from a call to llama_max_devices:

llama-cpp-python/llama_cpp/llama_cpp.py

Line 124 in eb56ce2

LLAMA_MAX_DEVICES = _lib.llama_max_devices()

You have most likely built llama-cpp-python without GPU support.

[thiner]

Interesting, I built my docker container from code, the key step is
RUN CUDACXX=/usr/local/cuda-12/bin/nvcc CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip3 install -e .[server]
with the built-out container, I encountered the LLAMA_MAX_DEVICES = 1 issue.

I modified this line this morning, change it to RUN CUDACXX=/usr/local/cuda-12/bin/nvcc CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip3 install -e ., it works!

[abetlen]

Closing for now, will reopen if someone has an issue where max devices is 1 with a GPU enabled installation.