Skip to content

Commit 71401ba

Browse files
AtsuMiyaiAtsuMiyai
committed
add upd
1 parent 24dc435 commit 71401ba

25 files changed

+1119
-0
lines changed

‎README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,19 @@ We also provide the raw data exported from Weights & Biases for the detailed res
190190
- MMMU (mmmu)
191191
- MMMU Validation (mmmu_val)
192192
- MMMU Test (mmmu_test)
193+
- MMUPDBench (mmupdbench)
194+
- MMUPDBench Base (mmupdbench_base)
195+
- MMAADBench Base (mmaadbench_base)
196+
- MMIASDBench Base (mmiasdbench_base)
197+
- MMIVQDBench Base (mmivqdbench_base)
198+
- MMUPDBench Option (mmupdbench_option)
199+
- MMAADBench Option (mmaadbench_option)
200+
- MMIASDBench Option (mmiasdbench_option)
201+
- MMIVQDBench Option (mmivqdbench_option)
202+
- MMUPDBench Instruction (mmupdbench_instruction)
203+
- MMAADBench Instruction (mmaadbench_instruction)
204+
- MMIASDBench Instruction (mmiasdbench_instruction)
205+
- MMIVQDBench Instruction (mmivqdbench_instruction)
193206
- MMVet (mmvet)
194207
- Multi-DocVQA (multidocvqa)
195208
- Multi-DocVQA Validation (multidocvqa_val)

‎lmms_eval/tasks/mmupdbench/_default_template_mmaadbench_base_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\n"
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmaad_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmaadbench_instruction_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nIf all the options are incorrect, answer \"F. None of the above\"."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmaad_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmaadbench_option_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nAnswer with the option's letter from the given choices directly."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmaad_option
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmiasdbench_base_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\n"
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmiasd_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmiasdbench_instruction_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nIf all the options are incorrect, answer \"F. None of the above\"."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmiasd_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmiasdbench_option_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nAnswer with the option's letter from the given choices directly."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmiasd_option
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmivqdbench_base_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\n"
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmivqd_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmivqdbench_instruction_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nIf the given image is irrelevant to the question, answer \"F. The image and question are irrelevant.\"."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmivqd_base
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

‎lmms_eval/tasks/mmupdbench/_default_template_mmivqdbench_option_yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
dataset_path: MM-UPD/MM-UPD
2+
doc_to_target: "answer"
3+
model_specific_prompt_kwargs:
4+
default:
5+
pre_prompt: ""
6+
post_prompt: "\nAnswer with the option's letter from the given choices directly."
7+
doc_to_visual: !function utils.mmupdbench_doc_to_visual
8+
doc_to_text: !function utils.mmupdbench_doc_to_text
9+
doc_to_target: "answer"
10+
process_results: !function utils.mmupdbench_process_results
11+
model_specific_generation_kwargs:
12+
llava:
13+
image_aspect_ratio: original
14+
output_type: generate_until
15+
dataset_name: mmivqd_option
16+
generation_kwargs:
17+
until:
18+
- "ASSISTANT:"
19+
max_new_tokens: 1024
20+
temperature: 0
21+
top_p: 0
22+
num_beams: 1
23+
do_sample: false

0 commit comments

Comments
 (0)