How can I disable legacy processing in llava-next

[foreverpiano]

System Info

4.47.1

Who can help?

vision models: @amyeroberts, @qubvel

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

transformers/src/transformers/models/llava_next/modeling_llava_next.py

Lines 866 to 916 in 5cabc75

	if legacy_processing:
	logger.warning_once(
	"Expanding inputs for image tokens in LLaVa-NeXT should be done in processing. "
	"Please add `patch_size` and `vision_feature_select_strategy` to the model's processing config or set directly "
	"with `processor.patch_size = {{patch_size}}` and processor.vision_feature_select_strategy = {{vision_feature_select_strategy}}`. "
	"Using processors without these attributes in the config is deprecated and will throw an error in v4.50."
	)
	if input_ids.shape[1] != 1:
	inputs_embeds = inputs_embeds.to(image_features.dtype)
	inputs_embeds, attention_mask, position_ids, labels, _ = self._merge_input_ids_with_image_features(
	image_features,
	feature_lens,
	inputs_embeds,
	input_ids,
	attention_mask,
	position_ids,
	labels=labels,
	)
	cache_position = torch.arange(attention_mask.shape[1], device=attention_mask.device)
	else:
	# Retrieve the first layer to inspect the logits and mask out the hidden states
	# that are set to 0
	first_layer_past_key_value = past_key_values[0][0][:, :, :, 0]
	# Sum all dimensions of head_dim (-2) to avoid random errors such as: https://github.com/huggingface/transformers/pull/28032#issuecomment-1863691941
	batch_index, non_attended_tokens = torch.where(first_layer_past_key_value.float().sum(-2) == 0)
	# Get the target length
	target_length = input_ids.shape[1]
	past_length = first_layer_past_key_value.shape[-1]
	extended_attention_mask = torch.ones(
	(attention_mask.shape[0], past_length),
	dtype=attention_mask.dtype,
	device=attention_mask.device,
	)
	# Filter out only the tokens that can be un-attended, this can happen
	# if one uses Llava + Fused modules where the cache on the
	# first iteration is already big enough, or if one passes custom cache
	valid_indices = non_attended_tokens < extended_attention_mask.size(-1)
	new_batch_index = batch_index[valid_indices]
	new_non_attended_tokens = non_attended_tokens[valid_indices]
	# Zero-out the places where we don't need to attend
	extended_attention_mask[new_batch_index, new_non_attended_tokens] = 0
	attention_mask = torch.cat((extended_attention_mask, attention_mask[:, -target_length:]), dim=1)
	position_ids = torch.sum(attention_mask, dim=1).unsqueeze(-1) - 1
	cache_position = torch.arange(attention_mask.shape[1], device=attention_mask.device)[-target_length:]
	# TODO: @raushan retain only the new behavior after v4.47

Sample script

def main():
    args = parse_args()
    
    processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
    model = LlavaNextForConditionalGeneration.from_pretrained(
        "llava-hf/llava-v1.6-mistral-7b-hf",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        attn_implementation="flash_attention_2",
    ).to("cuda:0")
    
    setup_model_with_compression(model, args)
    
    url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
    image = Image.open(requests.get(url, stream=True).raw)
    
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": "What is shown in this image?"},
            ],
        },
    ]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(image, prompt, return_tensors="pt").to("cuda:0")
    
    output = model.generate(**inputs, max_new_tokens=100)
    print(processor.decode(output[0], skip_special_tokens=True))

Expected behavior

how does the legacy processing work? can I disable it ?

[LysandreJik]

cc @zucchini-nlp as well

[zucchini-nlp]

@foreverpiano the legacy processing is disabled at this moment with the official checkpoints. There is one false warning emitted though which will be removed in the next release

If you want to update your own llava checkpoint for the new processing logic, you can do so with

processor.vision_feature_select_strategy = model.config.vision_feature_select_strategy
processor.patch_size = model.config.vision_config.patch_size
model.config.image_seq_length = {{MIN_SEQ_LENGTH_ONE_IMAGE_TAKES_FOR_YOUR_ARCHITECTURE}}

[foreverpiano]

@zucchini-nlp Thanks for your reply. But when I use the above script, it actually runs the legacy processing. Is it about the transformer version?

pip info:

torch==2.4.0
torchvision==0.19.0
numpy==1.26.0
transformers==4.47.1
datasets==2.15.0
accelerate==1.2.1
urllib3==2.1.0
bitsandbytes==0.41.2
chardet==5.2.0
scipy==1.11.4
deepdiff==6.7.1
diffusers==0.8.0
sentencepiece==0.1.99
protobuf==4.25.1

jieba==0.42.1
fuzzywuzzy==0.18.0
rouge==1.0.1
IPython==8.18.1

# tools for installing flash-attn
packaging==24.1
ninja
triton==3.0.0
pytest==8.2.1
flash_attn

matplotlib

[zucchini-nlp]

@foreverpiano yes, there is once a warning emitted and that will be fixed when the legacy processing is removed in #34502. But it should not mess up much with the generation quality

[foreverpiano]

def main():
    args = parse_args()
    
    processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
    model = LlavaNextForConditionalGeneration.from_pretrained(
        "llava-hf/llava-v1.6-mistral-7b-hf",
        torch_dtype=torch.float16,
        low_cpu_mem_usage=True,
        attn_implementation="flash_attention_2",
    ).to("cuda:0")
    

    processor.vision_feature_select_strategy = model.config.vision_feature_select_strategy # 
    processor.patch_size = model.config.vision_config.patch_size # 

    setup_model_with_compression(model, args)
    
    url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
    image = Image.open(requests.get(url, stream=True).raw)
    
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": "What is shown in this image?"},
            ],
        },
    ]
    
    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(image, prompt, return_tensors="pt").to("cuda:0")
    
    output = model.generate(**inputs, max_new_tokens=100)
    print(processor.decode(output[0], skip_special_tokens=True))

Hi, I add this patch_size in my script, but it still goes into legacy processing. I don't know why we need legacy processing.

@zucchini-nlp thanks

[foreverpiano]

Here is the log. It went into legacy processing many times.

[zucchini-nlp]

We just released transformers v4.48 last week, and the new release has all the legacy code removed. Can you try updating with pip install transformers==4.48 ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Rocketknight1]

Closing since I think this is no longer relevant as of 4.48