[Model] Support Dots OCR

[ywang96]

Purpose

This PR adds support for rednote-hilab/dots.ocr. This model is currently supported via OOT registration but we might as well bring it into vLLM so that users don't need to set it up with additonal steps.

Most of the codes taken are from the model repo, but this PR also cleans up a few logics that are no longer needed since this model does not support video modality.

FIXES #24581

Co-authored-by @yinz-aizip

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[casper-hansen]

Thanks for taking a stab at upstreaming this @ywang96. We need more and better OCR models, and this would be a great step forward.

[ywang96]

Thanks for taking a stab at upstreaming this @ywang96. We need more and better OCR models, and this would be a great step forward.

@casper-hansen No problem. BTW there were still some performance issues I still need to debug, but correctness-wise, this branch should be actually ready to go with vllm serve rednote-hilab/dots.ocr --trust-remote-code.

[ywang96]

@yinz-aizip I made some changes to use vllm internal layers - could you help verify the correctness of this implementation (similarly to what you did in your PR)? Thanks!

[yinz-aizip]

Summary

This PR compares performance and evaluation metrics between two commits:
• Commit 96cc9bc
• Commit c830194

The model was served with the following configuration:

hf_model_path='/path/to/dots.ocr'
export CUDA_VISIBLE_DEVICES=4,5,6,7
vllm serve $hf_model_path \
    --host 127.0.0.1 \
    --port 8126 \
    --data-parallel-size 4 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.8 \
    --chat-template-content-format string \
    --served-model-name model \
    --trust-remote-code

Efficiency

Throughput was estimated by running 1,000 concurrent requests on a single image.
• Commit 96cc9bc... → 3980.84
• Commit c830194... → 3743.34

The newer commit is slightly slower, though the difference is relatively minor.

Effectiveness

Evaluated on OmniDocBenchmark (lower is better):
• Commit 96cc9bc...
• overall_EN: 0.12578
• overall_CH: 0.16556
• Commit c830194...
• overall_EN: 0.12412
• overall_CH: 0.16285

The newer commit shows a small improvement in accuracy across both English and Chinese benchmarks.

Conclusion

•	Efficiency: Slightly reduced in the newer commit.
•	Effectiveness: Marginally improved results on OmniDocBenchmark.

Overall, the trade-off seems acceptable, with minor throughput loss balanced by better benchmark performance.

[casper-hansen]

@ywang96 I tested this PR:

• Correctness issues fixed (no more endless generations)
• Performance issues also fixed, now scales with number of concurrent images.

1x H100 concurrency benchmark:

• 1.81 images/second for 16 images.
• 2.36 images/second for 30 images.

[casper-hansen]

One potential performance issue: When I pass in 30 images, I see the below message 30 times before images are actually processed. This seems to add a big latency for this model.

Fetching 1 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7869.24it/s]

[ywang96]

One potential performance issue: When I pass in 30 images, I see the below message 30 times before images are actually processed. This seems to add a big latency for this model.

Fetching 1 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7869.24it/s]

Hmm - FYI @DarkLight1337, I'm wondering if this has something to do with a out-of-tree processor (i.e one with trust_remote_code=True) inheriting from a HF one?

[jeejeelee]

We aslo need to add this model in https://github.com/vllm-project/vllm/blob/main/tests/models/registry.py

[jeejeelee]

Can we use MergedColumnParallelLinear here ?

[jeejeelee]

Maybe we can use QKVParallelLinear.

[jeejeelee]

One potential performance issue: When I pass in 30 images, I see the below message 30 times before images are actually processed. This seems to add a big latency for this model.

Fetching 1 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7869.24it/s]

Hmm - FYI @DarkLight1337, I'm wondering if this has something to do with a out-of-tree processor (i.e one with trust_remote_code=True) inheriting from a HF one?

I tested it with vision_language.py and didn't see similar output, which is a bit strange.

[casper-hansen]

@ywang96 @yinz-aizip is it possible to avoid --trust-remote-code? I believe this is the root cause of the high latency.

[ywang96]

I believe this is the root cause of the high latency.

@casper-hansen Yea I think so too - but I don't think it should be removed but instead we should cache the object that fetches the remote file (instead of doing it over and over) - This should not happen and I need to debug why this is happening 😅

[DarkLight1337]

The repeated loading issue should be fixed by #25341

[ywang96]

The correctness of this PR has been verified by our contact from rednote engineering team so I'm just going to turn on ready label for it.

[jeejeelee]

Overall LGTM, some improvements can be completed in subsequent PRs