[Bug]: InternVL bounding box prediction does not work

[MoritzLaurer]

Your current environment

The output of `python collect_env.py`

```text python collect_env.py

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.9.5 (default, Jun 4 2021, 12:28:51) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.5.82
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.161.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.2.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7R32
Stepping: 0
CPU MHz: 3110.497
BogoMIPS: 5599.58
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 768 KiB
L1i cache: 768 KiB
L2 cache: 12 MiB
L3 cache: 96 MiB
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
[conda] nvidia-ml-py 12.560.30 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.6.77 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
[conda] pyzmq 26.2.0 pypi_0 pypi
[conda] torch 2.4.0 pypi_0 pypi
[conda] torchvision 0.19.0 pypi_0 pypi
[conda] transformers 4.45.2 pypi_0 pypi
[conda] triton 3.0.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.dev238+ge2c6e0a82
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-47 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks```

Model Input Dumps

No response

🐛 Describe the bug

I'm starting a vllm container with OpenGVLab/InternVL2 and vLLM version 0.6.2 with
vllm serve OpenGVLab/InternVL2-1B --trust-remote-code

As a prompt, I'm specifically using the object detection prompt: Please provide the bounding box coordinates of the region this sentence describes: <ref>{}</ref>. This prompt is documented here, here, or here. This prompt is designed to make the model output bounding box coordinates.

This prompt works correctly when used in transformers:

path = 'OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

# set the max number of tiles in `max_num`
pixel_values = load_image('./data/image.png', max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=True)

# single-image single-round conversation (单图单轮对话)
question = "<image>\nPlease provide the bounding box coordinate of the region this sentence describes: <ref>the top right cloud</ref>"
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')

#output: 
User: <image>
Please provide the bounding box coordinate of the region this sentence describes: <ref>the top right cloud</ref>
Assistant: the top right cloud[[619, 165, 958, 412]]

When I use the same standard bounding box prompt with vLLM and the OAI client, however, the output is gibberish:

from openai import OpenAI
import os
from time import time

# initialize the client but point it to TGI
client = OpenAI(
    base_url="http://localhost:8000/v1", 
    api_key="EMPTY"
)

generation_parameters = {
    "temperature": 0.2,
    "max_tokens": 128,
    "top_p": 0.7,
    #"max_input_tiles": 12,
    #"repetition_penalty": 1.2,
    "stream": False,
}

def predict(messages):
    return client.chat.completions.create(
        model="OpenGVLab/InternVL2-1B", # needs to be /repository since there are the model artifacts stored
        messages=messages,
        **generation_parameters
    ).choices[0].message.content


messages = [
    {"role": "system", 
     "content": "You are a helpful assistant",
    },
    {"role": "user", 
     "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://unsplash.com/photos/ZVw3HmHRhv0/download?ixid=M3wxMjA3fDB8MXxhbGx8NHx8fHx8fDJ8fDE3MjQ1NjAzNjl8&force=true&w=1920"
            }
        },
        {
            "type": "text",
            #"text": "What type of animal is in the image above? What color does it have?"
            "text": "\nPlease provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref>"  
        }
    ]},
]

response = predict(messages)
print(f"LLM output: {response}")

LLM output:  erótka exilis

The expected output would be something like the bird[[123, 234, 123, 234]] with bounding box coordinates for the object specified in the prompt.

The vLLM container does successfully generate text when prompted without this bounding-box prompt. e.g. the prompt "What type of animal is in the image above? What color does it have?" leads to this output with this image:
The animal in the image is a species of bird. The bird in the image is a Blue Tit. The bird has a distinctive and colorful plumage, with a blue head and a yellow and a bit of a red or orange throat.

I have tested this with several images and prompt variants and generation parameters. The bounding box prompt always outputs some form of gibberish, while other prompts output correct text. I also noticed that non-bounding box prompts produce correct text, but it also seems a bit suboptimal with repetitions and small errors.

I have a suspicion that there is something wrong in the chat template application done by vLLM for OpenGVLab/InternVL2. My assumption is that, because the model has only been trained with this bounding box prompt with the very specific chat template, it fully fails when there is a small deviation from the chat template. For normal prompts, the negative effect of the chat template issue might be less visible, because it probably has seen a wider variety of input formats, so it still produces correct, but suboptimal outputs. But this is just an assumption and the issue might come from somewhere else. Optimal InternVL outputs can also be created via this official HF Space demo for comparison.
I found the PR that added InternVL to vLLM, but I'm not sure which part of the code implements the preprocessor with the chat template.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Currently, we only support inserting image tokens (<image> etc.) per image, and hardcode it by model. This type of custom chat template where other tokens are inputted per image isn't supported in vLLM yet.

[DarkLight1337]

@ywang96 together with #7905 (comment), this is another case where we need to override how the image tokens are inserted per image input. We may have to work on this soon.

[Isotr0py]

This is odd, because InternVL2-2B can create correct bounding box result:

$ python examples/offline_inference_vision_language.py -m internvl_chat --num-prompts 1
WARNING 10-09 16:50:14 config.py:352] Async output processing is only supported for CUDA or TPU. Disabling it for other platforms.
INFO 10-09 16:50:14 llm_engine.py:237] Initializing an LLM engine (v0.6.1.post3.dev210+g83caf35e.d20241003) with config: model='/data/LLM-model/InternVL2-2B', speculative_config=None, tokenizer='/data/LLM-model/InternVL2-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/data/LLM-model/InternVL2-2B, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs={'max_dynamic_patch': 1})
WARNING 10-09 16:50:14 tokenizer.py:166] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
WARNING 10-09 16:50:14 cpu_executor.py:328] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 10-09 16:50:14 cpu_executor.py:354] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 10-09 16:50:14 selector.py:183] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 10-09 16:50:14 selector.py:128] Using Torch SDPA backend.
INFO 10-09 16:50:17 selector.py:183] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 10-09 16:50:17 selector.py:128] Using Torch SDPA backend.
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 71.83it/s]

INFO 10-09 16:50:19 cpu_executor.py:212] # CPU blocks: 2730
<s><|im_start|>user
<image>
Please provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref><|im_end|>
<|im_start|>assistant

WARNING 10-09 16:50:21 tokenizer.py:166] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:12<00:00, 12.95s/it, est. speed input: 22.63 toks/s, output: 1.47 toks/s]
the bird[[44, 409, 474, 936]]

However, both InternVL-1B and InternVL-4B indeed create gibberish outputs for box bounding.

[MoritzLaurer]

Thanks for your quick responses!

@DarkLight1337 , I'm not sure if this really requires a special logic/implementation. The prompt Please provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref> is just a string and these <ref> tokens are just special tokens which the the tokenizer can handle. So as long as these string are inputted in the same way the model has seen them during training, it should just work natively afaik (but maybe I'm wrong, I lack knowledge of the internals of vLLM).

@Isotr0py , that's very interesting. I had only tried InternVL-1B, because deploying 2B gave me this error (although 1B had accepted DISABLE_SLIDING_WINDOW=true):

Exit code: 1. Reason:         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
    self.max_model_len = _get_and_verify_max_len(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
    raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 134, in from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 874, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 811, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
    self.max_model_len = _get_and_verify_max_len(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
    raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate.

[MoritzLaurer]

Interesting, I confirm that I also get bounding boxes with the 2B model. (when successfully deployed with DISABLE_SLIDING_WINDOW=false)

[MoritzLaurer]

I tested whether the issue comes from DISABLE_SLIDING_WINDOW, but I still get gibberish with the 1B model when changing to DISABLE_SLIDING_WINDOW=false.

(I noticed that internvl tokenizes images to many tokens (3k+ tokens for just one image and a short text prompt), so thought that it could maybe be linked to the sliding window making the model only pay attention to parts of the input)

[Isotr0py]

Seems that InternVL2-4B can also output the correct bounding box on GPU, previous gibberish outputs may relate to the ipex on CPU backend:

$ python examples/offline_inference_vision_language.py -m internvl_chat
WARNING 10-09 11:04:13 config.py:1646] Casting torch.bfloat16 to torch.float16.
INFO 10-09 11:04:24 llm_engine.py:237] Initializing an LLM engine (v0.1.dev2938+gdc4aea6) with config: model='OpenGVLab/InternVL2-4B', speculative_config=None, tokenizer='OpenGVLab/InternVL2-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=OpenGVLab/InternVL2-4B, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
INFO 10-09 11:04:25 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 10-09 11:04:25 selector.py:116] Using XFormers backend.
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
INFO 10-09 11:04:26 model_runner.py:1051] Starting to load model OpenGVLab/InternVL2-4B...
INFO 10-09 11:04:26 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 10-09 11:04:26 selector.py:116] Using XFormers backend.
INFO 10-09 11:04:26 weight_utils.py:242] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:03<00:03,  3.96s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:07<00:00,  3.50s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:07<00:00,  3.57s/it]

INFO 10-09 11:04:34 model_runner.py:1062] Loading model weights took 7.7740 GB
INFO 10-09 11:04:38 gpu_executor.py:122] # GPU blocks: 810, # CPU blocks: 682
INFO 10-09 11:04:38 gpu_executor.py:126] Maximum concurrency for 4096 tokens per request: 3.16x
INFO 10-09 11:04:40 model_runner.py:1385] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 10-09 11:04:40 model_runner.py:1389] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 10-09 11:04:55 model_runner.py:1513] Graph capturing finished in 14 secs.
Processed prompts: 100%|███████████████████████████████████████████████| 4/4 [00:10<00:00,  2.56s/it, est. speed input: 1312.32 toks/s, output: 10.15 toks/s]
the bird[[39, 407, 497, 909]]
the bird[[46, 411, 503, 914]]
the bird[[42, 406, 497, 935]]
the bird[[41, 408, 495, 935]]

[MoritzLaurer]

I also just got it to work with the 4b model

[kulievvitaly]

I have tryied to use OpenGVLab/InternVL2-Llama3-76B-AWQ for bbox prediction. I use image 1920x1080. It correctly answers in json format, but bbox coordinates are bad.

[aixuedegege]

Description:

I am experiencing an issue with the VLLM + InternVL models when attempting to extract titles from images （OCR task）. Specifically:

• Model: VLLM + InternVL 1B

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Incorrect results are returned.

• Model: VLLM + InternVL 8B

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Correct results are returned.

• Model: Transformers (loading the 1B model directly)

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Correct results are returned.

Steps to Reproduce:

1. Load the VLLM + InternVL 1B model and provide an image containing a title.
2. Observe the output.
3. Repeat the process with the VLLM + InternVL 8B model and the Transformers 1B model.

Expected Outcome: The 1B model should provide results consistent with the Transformers model and the 8B model.

Actual Outcome: The 1B model in the VLLM + InternVL setup produces incorrect results.

[DarkLight1337]

Description:

I am experiencing an issue with the VLLM + InternVL models when attempting to extract titles from images （OCR task）. Specifically:

• Model: VLLM + InternVL 1B

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Incorrect results are returned.

• Model: VLLM + InternVL 8B

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Correct results are returned.

• Model: Transformers (loading the 1B model directly)

  • Expected Behavior: Should accurately identify the title in the image.
  • Observed Behavior: Correct results are returned.

Steps to Reproduce:

1. Load the VLLM + InternVL 1B model and provide an image containing a title.
2. Observe the output.
3. Repeat the process with the VLLM + InternVL 8B model and the Transformers 1B model.

Expected Outcome: The 1B model should provide results consistent with the Transformers model and the 8B model.

Actual Outcome: The 1B model in the VLLM + InternVL setup produces incorrect results.

Can you provide an example of how you passed the prompt text and image to the model?

[aixuedegege]

ok thank you , here is my code:

use vllm

from vllm import LLM, SamplingParams
from PIL import Image
llm = LLM(model="/data/OpenGVLab/InternVL2-1B", max_model_len=2048, gpu_memory_utilization=0.3, max_num_seqs=1, trust_remote_code=True, )

# sampling_params = SamplingParams(temperature=1, max_tokens=128)
sampling_params = SamplingParams(
    best_of=1,
    temperature=1.0,
    top_p=1,
    top_k=1,
    # use_beam_search=False,
    max_tokens=128,
    presence_penalty=0,
    frequency_penalty=0,
)

fp = "/data/images/0031.jpg"
image = Image.open(fp)
outputs = llm.generate({
    "prompt": "USER: <image>\n图片中的标题是什么？\nASSISTANT:",
    "multi_modal_data": {"image": image},
}, sampling_params
)

print("~" * 30)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

and it output nothing

use huggingface transformers

# copied from https://huggingface.co/OpenGVLab/InternVL2-8B#inference-with-transformers
path = '/data/OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

# set the max number of tiles in `max_num`
fp = "/data/images/0031.jpg"
pixel_values = load_image(fp, max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=False)

question = '<image>\n图片中的标题是什么？'
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')

and it output is right :

User:
图片中的标题是什么？
Assistant: 图片中的标题是“皖西卫生职业学院附属医院（六安市第二人民医院）出院记录”。

this is the subimage of my input image:

with vllm, i tried some other SamplingParams, they were all failed.
Is there something wrong with me?

[Isotr0py]

@aixuedegege Your prompt format used in vLLM is wrong, you can refer to examples/offline_inference_vision_language.py for chat template application.

[aixuedegege]

got it. thank you so much.