[LogProbs]Enable prompt logprobs output and modify data transmission method for the online interface.

[qwes5s5]

Motivation

• Currently, only the offline generate() interface supports outputting prompt logprobs results; the online interfaces, specifically v1/chat/completions and v1/completions, do not support prompt logprobs output.
• The v1/chat/completions and v1/completions interfaces do not support outputting the default vocabulary size when logprobs is set to -1; this requires improvement.
• The current transmission path from the engine layer to the API layer does not support the transfer of tensor data, and involves unnecessary memory copies, resulting in low efficiency when transmitting large volumes of data.

Modifications

• Added support and validation for the prompt_logprobs parameter input across both interfaces.
• Implemented logic in both interfaces for processing prompt_logprobs results and constructing the corresponding PromptLogprobs response.
• Modified the logprobs related logic in both interfaces to support the case where logprobs is set to -1.
• Revised the data transmission logic between the engine and API layers to support Paddle.Tensor transfer, utilizing ForkingPickler serialization and achieving zero-copy transfer.

Usage or Command

curl --location 'http://yq01-sys-rpm36jsah8z.yq01.baidu.com:8180/v1/completions' \
--header 'Content-Type: application/json' \
--data '{
  "prompt": "请生成一篇关于理想的作文,不超过100字",
  "logprobs":-1,
  "prompt_logprobs":5,
  "stream": "true",
  "stream_options":{
    "include_usage":"true"
  }
}'

curl --location 'http://yq01-sys-rpm36jsah8z.yq01.baidu.com:8180/v1/completions' \
--header 'Content-Type: application/json' \
--data '{
  "prompt": "请生成一篇关于理想的作文,不超过100字",
  "logprobs":5,
  "prompt_logprobs":5
}'

curl --location 'http://yq01-sys-rpm36jsah8z.yq01.baidu.com:8180/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {
            "role": "system",
            "content": "I'\''m a helpful AI assistant."
        },
        {
            "role": "user",
            "content": "give me three letter randdomly, just tell me letter without anything else"
        }
    ],
    "logprobs":"true",
    "top_logprobs":5,
    "prompt_logprobs":5
}'

curl --location 'http://yq01-sys-rpm36jsah8z.yq01.baidu.com:8180/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {
            "role": "system",
            "content": "I'\''m a helpful AI assistant."
        },
        {
            "role": "user",
            "content": "give me three letter randdomly, just tell me letter without anything else"
        }
    ],
    "logprobs":"true",
    "top_logprobs":5,
    "prompt_logprobs":5,
    "stream":true
}'

Accuracy Tests

When streaming output, the prompt_logprobs are included only in the first package (or chunk).

Checklist

• Add at least a tag in the PR title.
  • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
  • You can add new tags based on the PR content, but the semantics must be clear.
• Format your code, run pre-commit before commit.
• Add unit tests. Please write the reason in this PR if no unit tests.
• Provide accuracy results.
• If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

[paddle-bot]

Thanks for your contribution!

[qwes5s5]

/re-run run_tests_with_coverage

[qwes5s5]

/re-run run_ce_cases

[qwes5s5]

/re-run run_ce_cases

[gongshaotian]

LGTM for worker

[qwes5s5]

/re-run run_ce_cases

[qwes5s5]

/re-run run_tests_with_coverage

[Copilot]

Pull Request Overview

This pull request adds support for prompt_logprobs output in online interfaces (v1/chat/completions and v1/completions), implements logprobs=-1 to output default vocabulary size, and optimizes data transmission between engine and API layers using ForkingPickler with zero-copy transfer.

• Added prompt_logprobs parameter support with validation across both completion and chat interfaces
• Implemented logprobs=-1 functionality to return all vocabulary logprobs
• Replaced msgpack serialization with ForkingPickler for efficient Paddle.Tensor transfer
• Added extensive test coverage for new functionality

Reviewed Changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
tests/woker/test_gpu_prompt_logprobs.py	Added environment variable patching for FD_USE_GET_SAVE_OUTPUT_V1
tests/utils/test_clamp_prompt_logprobs.py	New tests for clamp_prompt_logprobs utility (tests expect AttributeError)
tests/entrypoints/test_vllm_run_engine.py	Updated tests for new validation logic with environment variable handling
tests/entrypoints/test_engine_client.py	Comprehensive validation tests for prompt_logprobs and max_logprobs
tests/entrypoints/openai/test_serving_completion.py	Extensive tests for prompt_logprobs in stream and full generators
tests/entrypoints/openai/test_serving_chat.py	Extensive tests for prompt_logprobs in chat completion generators
tests/engine/test_sampling_params.py	Updated validation tests with environment variable context
fastdeploy/utils.py	Added clamp_prompt_logprobs function (has immutability bug)
fastdeploy/worker/output.py	Updated PromptLogprobs type definition and added no_grad context
fastdeploy/inter_communicator/zmq_server.py	Changed from msgpack to ForkingPickler with zero-copy
fastdeploy/entrypoints/openai/utils.py	Updated deserialization to use ForkingPickler
fastdeploy/entrypoints/openai/serving_completion.py	Added _build_prompt_logprobs method and integration logic
fastdeploy/entrypoints/openai/serving_chat.py	Added _build_prompt_logprobs method (duplicated from completion)
fastdeploy/entrypoints/openai/protocol.py	Added prompt_logprobs fields and validation to request/response models
fastdeploy/entrypoints/engine_client.py	Added comprehensive validation for logprobs parameters
fastdeploy/entrypoints/llm.py	Updated validation logic for prompt_logprobs with vocab size handling
fastdeploy/engine/sampling_params.py	Conditional validation based on FD_USE_GET_SAVE_OUTPUT_V1
fastdeploy/config.py	Added max_logprobs validation against vocabulary size

[codecov-commenter]

Codecov Report

❌ Patch coverage is 67.56757% with 72 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@1372d6d). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/entrypoints/engine_client.py	7.69%	44 Missing and 4 partials ⚠️
fastdeploy/entrypoints/openai/protocol.py	65.51%	5 Missing and 5 partials ⚠️
fastdeploy/entrypoints/llm.py	68.75%	2 Missing and 3 partials ⚠️
fastdeploy/config.py	40.00%	1 Missing and 2 partials ⚠️
fastdeploy/inter_communicator/zmq_server.py	0.00%	3 Missing ⚠️
...astdeploy/entrypoints/openai/serving_completion.py	95.83%	2 Missing ⚠️
fastdeploy/entrypoints/openai/serving_chat.py	97.67%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #5089   +/-   ##
==========================================
  Coverage           ?   60.69%           
==========================================
  Files              ?      320           
  Lines              ?    39230           
  Branches           ?     5912           
==========================================
  Hits               ?    23812           
  Misses             ?    13530           
  Partials           ?     1888           

Flag	Coverage Δ
GPU	60.69% <67.56%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sunlei1024]

prompt_logprobs_tensors 无需重复定义，复用prompt_logprobs 即可，改成复合类型 [list | Tensor]

[Jiang-Jia-Jun]

os.getenv("FD_USE_GET_SAVE_OUTPUT_V1", "0"))这个改为使用envs.FD_USE_GET_SAVE_OUTPUT_V1，这样后面如需修改默认值，只需修改fastdeploy/env.py