Skip to content

Commit 11a6fc5

Browse files
abetlenDon Mahurin
authored and
Don Mahurin
committed
Allow model to tokenize strings longer than context length and set add_bos. Closes ggml-org#92
1 parent ac86ac0 commit 11a6fc5

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

examples/llama_cpp.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,7 @@ def llama_tokenize(
337337
tokens, # type: Array[llama_token]
338338
n_max_tokens: c_int,
339339
add_bos: c_bool,
340-
) -> c_int:
340+
) -> int:
341341
return _lib.llama_tokenize(ctx, text, tokens, n_max_tokens, add_bos)
342342

343343

0 commit comments

Comments
 (0)