kv_override issue

[sudharshan1234]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

To override values by passing the kv_override parameter as {"tokenizer.ggml.pre": "llama3"}
I expected llama-cpp-python to correctly handle the dictionary with values and override the values properly.

Current Behavior

Instead, I received an error "Received error 'bytes' object does not support item assignment (type=value_error)" when passing a dictionary with string value ("llama3") in this case.

Environment and Context

Python 3.8.1

• Physical (or virtual) hardware you are using, e.g. for Linux:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6246R CPU @ 3.40GHz
Stepping: 7
CPU MHz: 3999.999
BogoMIPS: 6800.00
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 36608K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31

• Operating System, e.g. for Linux:

Linux emul318 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

• SDK version, e.g. for Linux:

$ make --version
GNU Make 4.2.1
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.


$ g++ --version
g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Failure Information (for bugs)

Steps to Reproduce

1. Create a dictionary, e.g., kv_overrides = {"tokenizer.ggml.pre": "llama3"}
2. Pass this on to the Llama constructor

Failure Logs

Received error 'bytes' object does not support item assignment (type=value_error)

Findings

From the documentation, I think the error lies in the line

self._kv_overrides_array[i].value.str_ value[:128] = v_bytes

Here, self._kv_oversrides_array[i].value.str_value is a ctypes array of bytes (ctypes.ARRAY(ctypes.c_char, 128)), and you're trying to assign a bytes object to a slice of it.
https://llama-cpp-python.readthedocs.io/en/stable/api-reference/#llama_cpp.llama_cpp.llama_model_kv_override_value

In Python, bytes objects are immutable, so you can't assign to their elements individually. You need to create a new bytes object with the desired values.

[abetlen]

@sudharshan1234 thanks for pointing this out, will fix!