[Bugfix] Fix hermes tool parser handling of non-string argument types #22002

david6666666 · 2025-07-31T08:37:56Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Fix hermes tool parser handling of non-string argument types.

One example of the situation when argument type is integer is described in #21372.

Test Plan

Test the issue described in #21372.

Serving command:

vllm serve /workspace/models/Qwen3-4B     --reasoning-parser qwen3 --served-model-name 'Qwen/Qwen3-4B' --enable-auto-tool-choice --tool-call-parser hermes

Code:

import json

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

messages = [
    {
        "role":
        "system",
        "content":
        "You are an artificial intelligence assistant who will call tools everytime when responding.",
    },
    {
        "role":
        "user",
        "content":
        "Hi! Do you have any detailed information about the product id 7355608 and inserted true?",
    },
]
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_product_info",
            "description":
            "Get detailed information of a product based on its product ID.",
            "parameters": {
                "type": "object",
                "properties": {
                    "inserted": {
                        "type": "boolean",
                        "description": "inserted.",
                    },
                    "product_id": {
                        "type": "integer",
                        "description": "The product ID of the product.",
                    },
                },
                "required": ["product_id", "inserted"],
            },
        },
    },
]
use_stream = True
model = client.models.list().data[0].id
chat_completion = client.chat.completions.create(
    stream=use_stream,
    messages=messages,
    top_p=0.95,
    temperature=0.66,
    presence_penalty=0,
    frequency_penalty=0.04,
    model=model,
    tools=tools,
    extra_body={
        "top_k": 20,
        "repetition_penalty": 1.05,
        "chat_template_kwargs": {
            "enable_thinking": False
        },
    },
)

debug_list = list()
if use_stream:
    print("Tool call args:")
    for c in chat_completion:
        if c.choices[0].delta.tool_calls:
            print(c.choices[0].delta.tool_calls[0].function.arguments, end="")
        debug_list.append(c)
else:
    print("Chat completion tool calls:")
    print(chat_completion.choices[0].message.tool_calls)
    print(chat_completion.choices[0].message.tool_calls[0].function.arguments)
print("\n")

Test Result

use_stream = True

Tool call args:
None{"product_id": 7355608, "inserted": true}

use_stream = False

Chat completion tool calls:
[ChatCompletionMessageFunctionToolCall(id='chatcmpl-tool-67eb1b45d3b3474bae4bfde6c2fe20a2', function=Function(arguments='{"product_id": 7355608, "inserted": true}', name='get_product_info'), type='function')]
{"product_id": 7355608, "inserted": true}

gemini-code-assist

Code Review

This pull request fixes a bug in the Hermes tool parser where it failed to handle non-string argument types during streaming. The change correctly identifies whether the last argument is a string by inspecting the JSON string representation and adjusts how the string is processed. My review includes one suggestion to enhance the readability and maintainability of this critical but complex logic.

gemini-code-assist · 2025-07-31T08:40:09Z

vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py

+                stripped_cur_arguments_json = cur_arguments_json[:-2] \
+                    if (cur_arguments_json[-2] == '"'
+                            or cur_arguments_json[-2] == "'") else \
+                    cur_arguments_json[:-1]


While this logic correctly fixes the issue with non-string arguments, the ternary expression with a backslash for line continuation is a bit dense and can be hard to read and maintain.

For better clarity and maintainability, I suggest refactoring this into a standard if/else block. This makes the logic more explicit and easier to understand at a glance, which is valuable for complex string manipulations like this.

if cur_arguments_json[-2] in ('"', "'"): # Last argument is a string, so remove the closing quote and brace. stripped_cur_arguments_json = cur_arguments_json[:-2] else: # Last argument is not a string, so remove the closing brace only. stripped_cur_arguments_json = cur_arguments_json[:-1]

github-actions · 2025-07-31T09:39:37Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

david6666666 · 2025-07-31T11:49:49Z

@DarkLight1337 @chaunceyjiang please review, thanks

david6666666 · 2025-08-04T01:35:19Z

@aarnphm please review, thanks

chaunceyjiang

Can you add some unit tests?

chaunceyjiang · 2025-08-04T13:30:54Z

I used the test script you provided and the latest code from the main branch, but I couldn’t reproduce the issue you described.

Tool call args:
ChoiceDeltaToolCall(index=0, id='chatcmpl-tool-b843a3616a2a46ecb1a9dc2434713c3c', function=ChoiceDeltaToolCallFunction(arguments=None, name='get_product_info'), type='function')
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"product_id": 735', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='6', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='0', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='8', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='}', name=None), type=None)

BruceW-07 · 2025-08-05T00:52:29Z

I used the test script you provided and the latest code from the main branch, but I couldn’t reproduce the issue you described.

Tool call args:
ChoiceDeltaToolCall(index=0, id='chatcmpl-tool-b843a3616a2a46ecb1a9dc2434713c3c', function=ChoiceDeltaToolCallFunction(arguments=None, name='get_product_info'), type='function')
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"product_id": 735', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='6', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='0', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='8', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='}', name=None), type=None)

I think this is exactly the result described in the issue. The correct result should be "product_id": 7355608 (the number 5 should appear twice in the id)

chaunceyjiang · 2025-08-05T03:16:02Z

[Bugfix] Fix hermes tool parser handling of non-string argument types

@BruceW-07 My question is: in your PR description, you mentioned that the Hermes tool parser cannot handle non-string argument types, but based on my tests, "product_id": 735608 is a non-string argument and it seems to be processed correctly.

chaunceyjiang · 2025-08-05T03:19:49Z

I think this is exactly the result described in the issue. The correct result should be "product_id": 7355608 (the number 5 should appear twice in the id)

Do you mean that when the argument type is non-string, the Hermes tool parser unexpectedly removes some characters?

BruceW-07 · 2025-08-05T03:25:29Z

I think this is exactly the result described in the issue. The correct result should be "product_id": 7355608 (the number 5 should appear twice in the id)

Do you mean that when the argument type is non-string, the Hermes tool parser unexpectedly removes some characters?

Yes, and according to my observation, the original code did not take into account the case where the argument is non-string, which resulted in some characters being unexpectedly missing in the result.

chaunceyjiang · 2025-08-05T07:24:51Z

vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py

+                    stripped_cur_arguments_json = cur_arguments_json[:-2]
+                else:
+                    # last argument is not a string,
+                    #   so remove the closing brace only.


Can we add an example in the comment?

Sure! I added a simple example

chaunceyjiang · 2025-08-05T07:57:09Z

tests/tool_use/test_tool_calls.py

+    # validate arguments
+    streamed_args = json.loads(function_args_str)
+    assert isinstance(streamed_args, dict)
+    assert isinstance(streamed_args.get("product_id"), int)


Can you write another test with a bool parameter?

I believe bool types may also encounter the truncation issue with the tool parser.

No problem, I add a new Boolean argument to the function

openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) messages = [ { "role": "system", "content": "You are an artificial intelligence assistant who will call tools everytime when responding.", }, { "role": "user", "content": "Hi! Do you have any detailed information about the product id 7355608 and inserted true?", }, ] tools = [ { "type": "function", "function": { "name": "get_product_info", "description": "Get detailed information of a product based on its product ID.", "parameters": { "type": "object", "properties": { "inserted": { "type": "boolean", "description": "inserted.", }, "product_id": { "type": "integer", "description": "The product ID of the product.", }, }, "required": ["product_id", "inserted"], }, }, }, ] use_stream = True model = client.models.list().data[0].id chat_completion = client.chat.completions.create( stream=use_stream, messages=messages, top_p=0.95, temperature=0.66, presence_penalty=0, frequency_penalty=0.04, model=model, tools=tools, extra_body={ "top_k": 20, "repetition_penalty": 1.05, "chat_template_kwargs": { "enable_thinking": False }, }, )

I can reproduce the issue of the boolean value being truncated using this example.

Thanks, I've added it to the unit test

vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py

din0s · 2025-08-05T14:24:17Z

Here's another example which might be good to test for:

{
    "type": "function",
    "function": {
        "name": "search",
        "parameters": {
            "type": "object",
            "properties": {
                "search_request": {
                    "type": "object",
                    "properties": {
                        "query": {
                            "type": "string"
                        },
                        "retrieval_method": {
                            "enum": ["keyword", "neural", "rrf"],
                            "type": "string"
                        }
                    },
                    "required": ["query", "retrieval_method"]
                }
            },
            "required": ["search_request"]
        }
    }
}

Expected:
{"search_request": {"query": "latest transformers papers", "retrieval_method": "rrf"}}
Actual:
{"search_request": "latest transformers papers", "retrieval_method": "rrf"}

artmzhuk · 2025-08-06T09:08:54Z

Hi! Is there an ETA for this bugfix?

BruceW-07 · 2025-08-25T03:43:34Z

Here's another example which might be good to test for:

{
    "type": "function",
    "function": {
        "name": "search",
        "parameters": {
            "type": "object",
            "properties": {
                "search_request": {
                    "type": "object",
                    "properties": {
                        "query": {
                            "type": "string"
                        },
                        "retrieval_method": {
                            "enum": ["keyword", "neural", "rrf"],
                            "type": "string"
                        }
                    },
                    "required": ["query", "retrieval_method"]
                }
            },
            "required": ["search_request"]
        }
    }
}

Expected: {"search_request": {"query": "latest transformers papers", "retrieval_method": "rrf"}} Actual: {"search_request": "latest transformers papers", "retrieval_method": "rrf"}

Thanks for your example, I've fixed the issue and now I can get the correct result with Qwen3-4B. I haven't added it to the unit test yet, because the tool_chat_template_hermes.jinja used in the test seems to have problem dealing with the example you provided.

BruceW-07 · 2025-08-25T03:44:06Z

@aarnphm @chaunceyjiang please review, thanks!

Signed-off-by: wangzi <[email protected]>

Signed-off-by: David Chen <[email protected]>

gcalmettes · 2025-09-19T07:26:58Z

vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py

+                    r'\{"name":\s*"' +
+                    re.escape(function_name) + r'"\s*,\s*"arguments":\s*(.*)',
+                    tool_call_portion.strip(), re.DOTALL)
+                cur_arguments_json = match.group(1)


There might still be a need to check if there is a match, as the match could be None for string arguments:

(APIServer pid=1) DEBUG 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:344] diffing old arguments: {} (APIServer pid=1) DEBUG 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:345] against new ones: {'name': '263012.pdf'} (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] Error trying to handle streaming tool call. (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] Traceback (most recent call last): (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 375, in extract_tool_calls_streaming (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] cur_arguments_json = match.group(1) (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] ^^^^^^^^^^^ (APIServer pid=1) ERROR 09-19 07:20:26 [entrypoints/.../tool_parsers/hermes_tool_parser.py:441] AttributeError: 'NoneType' object has no attribute 'group'

adding this change fixed it in my case:

if match: cur_arguments_json = match.group(1) else: cur_arguments_json = json.dumps(cur_arguments, ensure_ascii=False)

yes, thanks for your suggestion

Signed-off-by: David Chen <[email protected]>

…vllm-project#22002) Signed-off-by: wangzi <[email protected]> Signed-off-by: David Chen <[email protected]> Co-authored-by: wangzi <[email protected]> Co-authored-by: Chauncey <[email protected]>

…vllm-project#22002) Signed-off-by: wangzi <[email protected]> Signed-off-by: David Chen <[email protected]> Co-authored-by: wangzi <[email protected]> Co-authored-by: Chauncey <[email protected]> Signed-off-by: charlifu <[email protected]>

…#22002) Signed-off-by: wangzi <[email protected]> Signed-off-by: David Chen <[email protected]> Co-authored-by: wangzi <[email protected]> Co-authored-by: Chauncey <[email protected]> Signed-off-by: yewentao256 <[email protected]>

…vllm-project#22002) Signed-off-by: wangzi <[email protected]> Signed-off-by: David Chen <[email protected]> Co-authored-by: wangzi <[email protected]> Co-authored-by: Chauncey <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>

…vllm-project#22002) Signed-off-by: wangzi <[email protected]> Signed-off-by: David Chen <[email protected]> Co-authored-by: wangzi <[email protected]> Co-authored-by: Chauncey <[email protected]>

david6666666 requested a review from aarnphm as a code owner July 31, 2025 08:37

mergify bot added frontend tool-calling labels Jul 31, 2025

github-project-automation bot added this to Tool Calling Jul 31, 2025

gemini-code-assist bot reviewed Jul 31, 2025

View reviewed changes

chaunceyjiang reviewed Aug 4, 2025

View reviewed changes

chaunceyjiang reviewed Aug 5, 2025

View reviewed changes

vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py Outdated Show resolved Hide resolved

chaunceyjiang mentioned this pull request Aug 12, 2025

[Bugfix] hermes解析器输出存在被过滤问题 #22729

Open

4 tasks

david6666666 force-pushed the bugfix-21372-wz branch from 3c870a2 to b7ddfec Compare August 26, 2025 01:43

david6666666 requested a review from chaunceyjiang August 26, 2025 09:18

BruceW-07 added 6 commits September 2, 2025 16:06

[Bugfix] Fix hermes tool parser handling of non-string argument types

5c08420

Signed-off-by: wangzi <[email protected]>

Refactor the code for better clarity

bc4b1c3

Signed-off-by: wangzi <[email protected]>

add unit test for tool calls with integar argument

96fec13

Signed-off-by: wangzi <[email protected]>

add an example in the comment

2cdb3e3

Signed-off-by: wangzi <[email protected]>

add an example in the comment

7e27d09

Signed-off-by: wangzi <[email protected]>

add test for boolean arguments

e927b49

Signed-off-by: wangzi <[email protected]>

david6666666 requested review from alexm-redhat and comaniac as code owners September 19, 2025 06:59

mergify bot added ci/build deepseek Related to DeepSeek models llama Related to Llama models multi-modality Related to multi-modality (#4194) gpt-oss Related to GPT-OSS models speculative-decoding v1 labels Sep 19, 2025

github-project-automation bot added this to gpt-oss Issues & Enhancements Sep 19, 2025

mergify bot added the tpu Related to Google TPUs label Sep 19, 2025

github-project-automation bot moved this to To Triage in gpt-oss Issues & Enhancements Sep 19, 2025

david6666666 force-pushed the bugfix-21372-wz branch from 6b5b690 to 1a37f4f Compare September 19, 2025 07:00

mergify bot removed the tpu Related to Google TPUs label Sep 19, 2025

david6666666 added 2 commits September 19, 2025 15:06

fix test

8487305

Signed-off-by: David Chen <[email protected]>

fix test

28e7e6f

Signed-off-by: David Chen <[email protected]>

gcalmettes reviewed Sep 19, 2025

View reviewed changes

david6666666 added 3 commits September 19, 2025 15:27

fix pre-commit

9b43f14

Signed-off-by: David Chen <[email protected]>

fix the match could be None for string arguments

e25075d

Signed-off-by: David Chen <[email protected]>

fix the match could be None for string arguments

9271c2f

Signed-off-by: David Chen <[email protected]>

chaunceyjiang merged commit 0eecb31 into vllm-project:main Sep 22, 2025
43 checks passed

github-project-automation bot moved this from To Triage to Done in gpt-oss Issues & Enhancements Sep 22, 2025

github-project-automation bot moved this to Done in Tool Calling Sep 22, 2025

Uh oh!

[Bugfix] Fix hermes tool parser handling of non-string argument types #22002

[Bugfix] Fix hermes tool parser handling of non-string argument types #22002

Conversation

david6666666 commented Jul 31, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

david6666666 commented Jul 31, 2025

Uh oh!

david6666666 commented Aug 4, 2025

Uh oh!

chaunceyjiang left a comment

Choose a reason for hiding this comment

Uh oh!

chaunceyjiang commented Aug 4, 2025

Uh oh!

BruceW-07 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chaunceyjiang commented Aug 5, 2025

Uh oh!

chaunceyjiang commented Aug 5, 2025

Uh oh!

BruceW-07 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

din0s commented Aug 5, 2025

Uh oh!

artmzhuk commented Aug 6, 2025

Uh oh!

BruceW-07 commented Aug 25, 2025

Uh oh!

BruceW-07 commented Aug 25, 2025

Uh oh!

gcalmettes Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

david6666666 commented Jul 31, 2025 •

edited by github-actions bot

Loading

BruceW-07 commented Aug 5, 2025 •

edited

Loading

BruceW-07 commented Aug 5, 2025 •

edited

Loading

gcalmettes Sep 19, 2025 •

edited

Loading