bug: Cortex-cpp continues to have 1 layer offload to CPU while using GPU

[Van-QA]

Describe the bug
When generating responses using a local llm, cortex-cpp still seems to use CPU.
https://discord.com/channels/1107178041848909847/1149558035971321886/1253148982188838954

To Reproduce

1. Install cortex-cpp and the CUDA toolkit locally.
2. Turn on GPU acceler‌‌atio‌n
3. Generate responses using a local llm.
4. Observe high CPU usage.

Expected behavior
Since cortex-cpp is using a local llm and the CUDA toolkit, it should primarily use the GPU for processing and not consume as much CPU.

Desktop

• OS: Linux

Additional context
The logs indicate that 32 out of 33 layers are offloaded to the GPU, but 1 layer is still processed on the CPU. This behavior will be investigated further.

[dan-menlo]

@nguyenhoangthuan99 I am linking this to #1151 as a sub-issue. Please let me know if already solved

[vansangpfiev]

This should be resolved by changing ngl in model.yml

[gabrielle-ong]

Edit: set to the maximum ngl

[dan-menlo]

@hahuyhoang411 @gabrielle-ong I recommend we create a separate issue in Models repo, for our model.yaml to include num_layers:

• Allow for user to use the slider in Jan to offload to GPU (i.e. "Max")
• Allow for simple checking to prevent int out of bounds etc

This would solve for the issue where people are unsure how many layers are in a model, which results in slow inference due to layers being left on CPU instead of being fully offloaded to GPU