Guided grammar api signature inquiry

[rlucas7]

System Info

Lucass-MacBook:Desktop rlucas$ python -m "torch.utils.collect_env"
:128: RuntimeWarning: 'torch.utils.collect_env' found in sys.modules after import of package 'torch.utils', but prior to execution of 'torch.utils.collect_env'; this may result in unpredictable behaviour
Collecting environment information...
PyTorch version: 2.2.2
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.7.6 (x86_64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.21.1
Libc version: N/A

Python version: 3.11.6 (v3.11.6:8b6ee5ba3b, Oct 2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-13.7.6-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) m3-7Y32 CPU @ 1.10GHz

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.2
[conda] Could not collect

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

Hi I'm trying to determine if the guided_grammar option is supported by the llama4 model.

What I've tried:

pascal like EBNF cribbed from wikipedia

https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form

ebnf = '''
 program = 'PROGRAM', white space, identifier, white space, 
            'BEGIN', white space, 
            { assignment, ";", white space },
            'END.' ;
 identifier = alphabetic character, { alphabetic character | digit } ;
 number = [ "-" ], digit, { digit } ;
 string = '"' , { all characters - '"' }, '"' ;
 assignment = identifier , ":=" , ( number | identifier | string ) ;
 alphabetic character = "A" | "B" | "C" | "D" | "E" | "F" | "G"
                      | "H" | "I" | "J" | "K" | "L" | "M" | "N"
                      | "O" | "P" | "Q" | "R" | "S" | "T" | "U"
                      | "V" | "W" | "X" | "Y" | "Z" ;
 digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
 white space = ? white space characters ? ;
 all characters = ? all visible characters ? ;
 '''

import os
from llama_api_client import LlamaAPIClient

HERD = {"mav": "Llama-4-Maverick-17B-128E-Instruct-FP8", # groq ??
        "scout": "Llama-4-Scout-17B-16E-Instruct-FP8" # Cerebras??
        }

client = LlamaAPIClient()

code = """```python\n
          A = 3
          B = 45
          C = -100023
          D = A + B + C
          E = 'that is all for now'
          ```"""

completion = client.chat.completions.create(
    model=HERD["mav"],
    messages=[
        {
            "role": "user",
            "content": f"""Convert the following program written in python:\n\n\n
                          {code}"""
        }
    ],
       extra_body={
        "guided_grammar": ebnf
    }
)
print(completion.completion_message.content.text)

What I get back:

>>> print(completion.completion_message.content.text)
**Equivalent Code in Other Programming Languages**

The given Python code is actually not utilizing Python's syntax fully as it's not using the Pythonic way of writing code. Nevertheless, here's the equivalent code in a few other programming languages.

### Python (Improved Version)

```python
# Define variables
A = 3
B = 45
C = -100023

# Calculate D
D = A + B + C

# Print the result (optional)
print("The value of D is:", D)

Java

public class Main {
    public static void main(String[] args) {
        int A = 3;
        int B = 45;
        int C = -100023;
        int D = A + B + C;
        System.out.println("The value of D is: " + D);
    }
}

C

#include <stdio.h>

int main() {
    int A = 3;
    int B = 45;
    int C = -100023;
    int D = A + B + C;
    printf("The value of D is: %d\n", D);
    return 0;
}

C++

#include <iostream>

int main() {
    int A = 3;
    int B = 45;
    int C = -100023;
    int D = A + B + C;
    std::cout << "The value of D is: " << D << std::endl;
    return 0;
}

JavaScript

let A = 3;
let B = 45;
let C = -100023;
let D = A + B + C;
console.log("The value of D is:", D);

To convert the given Python code into another language, you need to follow these steps:

1. Identify the data types: In the given code, A, B, C, and D are integers.
2. Declare variables: Declare the variables with the same data type in the target language.
3. Perform the operation: Use the same arithmetic operation (+) to calculate D.
4. Output the result (optional): If needed, print or display the value of D.

These steps are demonstrated in the code snippets above for different programming languages.

So this isn't precisely what I want, what seems to work is to pass the EBNF directly like so:

```python
completion = client.chat.completions.create(
    model=HERD["mav"],
    messages=[
        {
            "role": "user",
            "content": f"""Convert the following program written in python:\n\n\n
                          {code}\n\nEBNF:\n{ebnf}"""
        }
    ]
)
print(completion.completion_message.content.text)

Giving me the output:

To convert the given Python program to the language defined by the provided EBNF (Extended Backus-Naur Form) grammar, we first need to understand the structure defined by the EBNF.

The given EBNF defines a simple programming language with the following key elements:
- A program starts with 'PROGRAM', followed by an identifier (the program name), 'BEGIN', and then a sequence of assignments ending with 'END.'.
- Assignments are in the form of `identifier := (number | identifier | string)`.

Let's analyze the given Python program and convert it according to the EBNF:

```python
A = 3
B = 45
C = -100023
D = A + B + C

1. Identify the Program Name: The Python code doesn't explicitly have a program name. We'll choose 'ExampleProgram' as our identifier.

2. Translate Assignments: We need to translate the Python assignments into the form defined by the EBNF.

   • A = 3 becomes A := 3
   • B = 45 becomes B := 45
   • C = -100023 becomes C := -100023
   • D = A + B + C needs to be broken down because the EBNF does not directly support expressions like A + B + C. However, we can achieve the same result by sequential assignments:
     • D := A
     • D := D + B
     • D := D + C

However, upon closer inspection, we see that the EBNF grammar doesn't directly support expressions like D + B or D + C on the right-hand side of an assignment. It only allows a single number, identifier, or string. To adhere strictly to the given EBNF, we cannot directly translate D = A + B + C.

For the sake of this exercise, let's assume we can slightly adjust our interpretation to allow for a more straightforward translation, focusing on the structure rather than the expression evaluation. In a real-world scenario, you would need to either modify the EBNF to support expressions or handle expressions in a more complex way (e.g., using a parser that can handle arithmetic expressions).

Here's a translation that follows the spirit of the EBNF but acknowledges the limitation regarding expressions:

PROGRAM ExampleProgram
BEGIN
    A := 3;
    B := 45;
    C := -100023;
    D := A;
    D := D + B;  // This line and the next are not directly supported by the given EBNF
    D := D + C;  // These would need additional handling or EBNF modification
END.

To strictly follow the EBNF as given, we can only assign a single number, identifier, or string to a variable. Thus, the expression D = A + B + C cannot be directly translated.

Let's adjust our translation to fit within the constraints of the provided EBNF, focusing on simple assignments:

PROGRAM ExampleProgram
BEGIN
    A := 3;
    B := 45;
    C := -100023;
    D := A;
END.

This version adheres to the EBNF by only using allowed constructs. For more complex operations, the language defined by the EBNF would need to be extended.

To implement a parser or translator for this EBNF, you would typically use a parsing technique such as recursive descent parsing, given the simplicity of the grammar. However, implementing a full parser is beyond the scope of this response.

It's also nice that it identifies the mismatch and explains the ambivalence between the Grammar and the written code. :)


Is there some other way we expect to pass in the grammar decoding type examples, or is the expectation that we add them in the prompt as I did in the second example?   

### Error logs

NA

### Expected behavior

NA, I'm asking which way is the expected call. 

[rlucas7]

swapping the ambiguous grammar with

ebnf = '''
 program = 'PROGRAM', white space, identifier, white space, 
            'BEGIN', white space, 
            { assignment, ";", white space },
            'END.' ;
 identifier = alphabetic character, { alphabetic character | digit } ;
 number = [ "-" ], digit, { digit } ;
 string = '"' , { all characters - '"' }, '"' ;
 assignment = identifier , ":=" , expression ;
 expression = term { ("+" | "-") term } ;
 term = factor { ("*" | "/") factor } ;
 factor = number | identifier | "(" , expression , ")" ;
 alphabetic character = "A" | "B" | "C" | "D" | "E" | "F" | "G"
                      | "H" | "I" | "J" | "K" | "L" | "M" | "N"
                      | "O" | "P" | "Q" | "R" | "S" | "T" | "U"
                      | "V" | "W" | "X" | "Y" | "Z" ;
 digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
 white space = ? white space characters ? ;
 all characters = ? all visible characters ? ;
 expression = term { ("+" | "-") term } ;
 '''

and passing directly in the prompt seems to give a reasonable answer to the translation.

[rlucas7]

https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/protocol.html#vllm.entrypoints.openai.protocol.ChatCompletionRequest for context

here is a direct link to the embedded line:

https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/protocol.html#__codelineno-0-349

[subramen]

@rlucas7 Guided Decoding an inference-time step, and can be done for any LLM if you can access the logits of the forward pass. Check out
[1] https://docs.vllm.ai/en/v0.8.2/features/structured_outputs.html
[2] https://dottxt-ai.github.io/outlines/latest/reference/generation/cfg/