Skip to content

llama: fix error on bad grammar #12628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

JohannesGaessler
Copy link
Collaborator

On master passing a bad grammar to either llama-cli or the server currently does not result in an error, there is only a print indicating that the grammar could not be parsed and the grammar is otherwise ignored. However, I don't think that this is the intended behavior. The code checks if the return value of common_sampler_init is null, it's just that the return value is not null when passing a bad grammar. This PR makes it so that llama_sampler_init_grammar_impl returns null when grammar-str is not empty but could not be passed, this is then propagated upwards to return null with common_sampler_init. When passing a bad grammar to the server the server now returns an error, llama-cli terminates with an error message.

I used the following commands for testing:

export model_name=llama_3-8b && export quantization=q4_0
./build/bin/llama-cli --model models/opt/${model_name}-${quantization}.gguf -ngl 99 --n-predict 1 --prompt "1 2 3 4 " --grammar "root ::= \"6\""
./build/bin/llama-cli --model models/opt/${model_name}-${quantization}.gguf -ngl 99 --n-predict 1 --prompt "1 2 3 4 " --grammar "asdf"
./build/bin/llama-server --model models/opt/${model_name}-${quantization}.gguf --n-gpu-layers 99
curl --request POST --url http://localhost:8080/completion --data '{"prompt": "1 2 3 4 ", "n_predict": 1, "grammar": "root ::= \"6\""}' | python3 -m json.tool
curl --request POST --url http://localhost:8080/completion --data '{"prompt": "1 2 3 4 ", "n_predict": 1, "grammar": "asdf"}' | python3 -m json.tool

@JohannesGaessler JohannesGaessler merged commit dd373dd into ggml-org:master Mar 28, 2025
51 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants