I'm unable to reproduce the results from the paper.

[ZZC-CN]

This is my configuration file.

and this is my results.

I want to get the result like this.

Is there an issue with my settings? Could you help me troubleshoot it?

[mbreuss]

Hi,
the results look pretty weird. Did you train the model yourself or which checkpoint did you use? Can you share the logs of the model loading? Did something go wrong there and only parts of the model have been loaded?

[ZZC-CN]

Thank you for your response.

1. I didn't finetune the model and i use the model in Figrue named mdtv-1-d

2.This is all my log

(mdt_env) zzc@zzc-System-Product-Name:~/mydata/mdt$ python mdt_policy/mdt/evaluation/mdt_evaluate.py
Global seed set to 142
0
[2025-03-26 09:56:27,671][mdt.datasets.base_dataset][INFO] - loading dataset at /home/zzc/mydata/mdt/task_D_D/training
[2025-03-26 09:56:27,672][mdt.datasets.base_dataset][INFO] - finished loading dataset
trying to load lang data from: /home/zzc/mydata/mdt/task_D_D/training/lang_annotations/auto_lang_ann.npy
[2025-03-26 09:56:27,902][mdt.datasets.base_dataset][INFO] - loading dataset at /home/zzc/mydata/mdt/task_D_D/validation
[2025-03-26 09:56:27,902][mdt.datasets.base_dataset][INFO] - finished loading dataset
trying to load lang data from: /home/zzc/mydata/mdt/task_D_D/validation/lang_annotations/auto_lang_ann.npy
[2025-03-26 09:56:27,965][mdt.datasets.base_dataset][INFO] - loading dataset at /home/zzc/mydata/mdt/task_D_D/training
[2025-03-26 09:56:27,965][mdt.datasets.base_dataset][INFO] - finished loading dataset
[2025-03-26 09:56:27,966][mdt.datasets.disk_dataset][INFO] - Found "ep_start_end_ids.npy" with 31 episodes.
[2025-03-26 09:56:27,994][mdt.datasets.base_dataset][INFO] - loading dataset at /home/zzc/mydata/mdt/task_D_D/validation
[2025-03-26 09:56:27,995][mdt.datasets.base_dataset][INFO] - finished loading dataset
[2025-03-26 09:56:27,995][mdt.datasets.disk_dataset][INFO] - Found "ep_start_end_ids.npy" with 4 episodes.
pybullet build time: Jan 29 2025 23:19:57
[2025-03-26 09:56:30,223][mdt.wrappers.hulc_wrapper][WARNING] - Couldn't find correct EGL device. Setting EGL_VISIBLE_DEVICE=0. When using DDP with many GPUs this can lead to OOM errors. Did you install PyBullet correctly? Please refer to calvin env README
[2025-03-26 09:56:30,223][mdt.wrappers.hulc_wrapper][INFO] - EGL_DEVICE_ID 0 <==> CUDA_DEVICE_ID 0
argv[0]=
[2025-03-26 09:56:30,311][calvin_env.envs.play_table_env][INFO] - Connected to server with id: 0
[2025-03-26 09:56:30,325][calvin_env.envs.play_table_env][INFO] - Connected to server with id: 0
[2025-03-26 09:56:30,363][calvin_env.robot.robot][INFO] - Loading robot
[2025-03-26 09:56:30,370][calvin_env.envs.play_table_env][INFO] - Resetting simulation
[2025-03-26 09:56:30,371][calvin_env.envs.play_table_env][INFO] - Setting gravity
WARNING uncommitted modified files: calvin_env/envs/play_table_env.py,conf/env/play_table_env.yaml
[2025-03-26 09:56:30,634][calvin_env.envs.play_table_env][INFO] - Using calvin_env with commit 797142c588c21e76717268b7b430958dbd13bf48.
[2025-03-26 09:56:30,634][mdt.wrappers.hulc_wrapper][INFO] - Initialized PlayTableEnv for device cuda:0
Loading model from /home/zzc/mydata/mdt/mdtv_1_d/saved_models/avg_seq_len=3.75-001.ckpt
class_name mdt.models.mdtv_agent.MDTVAgent
Loading model from /home/zzc/mydata/mdt/mdtv_1_d/saved_models/avg_seq_len=3.75-001.ckpt
[2025-03-26 09:57:07,538][py.warnings][WARNING] - /home/zzc/mydata/.conda/envs/mdt_env/lib/python3.8/site-packages/torchvision/datapoints/init.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)

[2025-03-26 09:57:07,554][py.warnings][WARNING] - /home/zzc/mydata/.conda/envs/mdt_env/lib/python3.8/site-packages/torchvision/transforms/v2/init.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: pytorch/vision#6753, and you can also check out pytorch/vision#7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().
warnings.warn(_BETA_TRANSFORMS_WARNING)

loading vision CLIP model with backbone: ViT-B/16
loading language CLIP model with backbone: ViT-B/32
obs dim: 384, goal_dim: 512, action_dim: 7, proprio_dim: 8
[2025-03-26 09:57:25,269][mdt.models.networks.mdtv_transformer][INFO] - Number of encoder parameters: 7.09
[2025-03-26 09:57:25,269][mdt.models.networks.mdtv_transformer][INFO] - Number of decoder parameters: 13.0
[2025-03-26 09:57:25,270][mdt.models.networks.mdtv_transformer][INFO] - number of parameters: 2.251777e+07
Successfully loaded EMA weights from checkpoint!
Finished loading model /home/zzc/mydata/mdt/mdtv_1_d/saved_models/avg_seq_len=3.75-001.ckpt
Successfully loaded model.
10 ddim 10 0.001 80 exponential
logging to /home/zzc/mydata/mdt/mdtv_1_d/logs/2025-03-26_09-57-27
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: 1055591303 (1055591303-northeastern-university) to https://api.wandb.ai. Use wandb login --relogin to force relogin
wandb: Tracking run with wandb version 0.19.8
wandb: Run data is saved locally in /home/zzc/mydata/mdt/mdtv_1_d/logs/2025-03-26_09-57-27/wandb/wandb/run-20250326_095728-3xxoe903
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run light-hill-10
wandb: ⭐️ View project at https://wandb.ai/1055591303-northeastern-university/calvin_eval
wandb: 🚀 View run at https://wandb.ai/1055591303-northeastern-university/calvin_eval/runs/3xxoe903
[2025-03-26 09:57:31,467][mdt.evaluation.multistep_sequences][INFO] - Start generating evaluation sequences.
[2025-03-26 09:57:38,871][mdt.evaluation.multistep_sequences][INFO] - Done generating evaluation sequences.
0%| | 0/1000 [00:00<?, ?it/s][2025-03-26 09:57:44,349][py.warnings][WARNING] - /home/zzc/mydata/.conda/envs/mdt_env/lib/python3.8/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
warnings.warn(

1/5 : 38.8% | 2/5 : 9.7% | 3/5 : 1.2% | 4/5 : 0.1% | 5/5 : 0.0% | Average: 0.5 |: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [5:10:05<00:00, 18.61s/it]
Results for Epoch 3.75-001:
Average successful sequence length: 0.498
Success rates for i instructions in a row:
1: 38.8%
2: 9.7%
3: 1.2%
4: 0.1%
5: 0.0%
turn_off_led: 52 / 67 | SR: 77.6%
push_into_drawer: 14 / 26 | SR: 53.8%
lift_pink_block_slider: 8 / 40 | SR: 20.0%
place_in_slider: 6 / 13 | SR: 46.2%
push_pink_block_right: 11 / 37 | SR: 29.7%
push_red_block_left: 15 / 48 | SR: 31.2%
turn_on_led: 68 / 72 | SR: 94.4%
close_drawer: 50 / 62 | SR: 80.6%
turn_on_lightbulb: 42 / 55 | SR: 76.4%
move_slider_right: 98 / 120 | SR: 81.7%
rotate_pink_block_right: 9 / 47 | SR: 19.1%
move_slider_left: 32 / 99 | SR: 32.3%
rotate_blue_block_left: 6 / 35 | SR: 17.1%
rotate_red_block_right: 5 / 40 | SR: 12.5%
push_pink_block_left: 10 / 46 | SR: 21.7%
turn_off_lightbulb: 16 / 46 | SR: 34.8%
open_drawer: 4 / 170 | SR: 2.4%
rotate_blue_block_right: 4 / 40 | SR: 10.0%
push_red_block_right: 7 / 46 | SR: 15.2%
rotate_red_block_left: 2 / 28 | SR: 7.1%
push_blue_block_left: 9 / 45 | SR: 20.0%
rotate_pink_block_left: 6 / 30 | SR: 20.0%
push_blue_block_right: 4 / 38 | SR: 10.5%
lift_pink_block_table: 10 / 47 | SR: 21.3%
lift_red_block_table: 4 / 43 | SR: 9.3%
place_in_drawer: 2 / 4 | SR: 50.0%
lift_blue_block_table: 2 / 43 | SR: 4.7%
lift_red_block_slider: 2 / 49 | SR: 4.1%
lift_blue_block_drawer: 0 / 3 | SR: 0.0%
lift_blue_block_slider: 0 / 49 | SR: 0.0%
stack_block: 0 / 9 | SR: 0.0%
lift_red_block_drawer: 0 / 1 | SR: 0.0%

Best model: epoch 3.75-001 with average sequences length of 0.498
wandb:
wandb:
wandb: Run history:
wandb: avrg_performance/avg_seq_len ▁
wandb:
wandb: Run summary:
wandb: avrg_performance/avg_seq_len 0.498
wandb:
wandb: 🚀 View run light-hill-10 at: https://wandb.ai/1055591303-northeastern-university/calvin_eval/runs/3xxoe903
wandb: ⭐️ View project at: https://wandb.ai/1055591303-northeastern-university/calvin_eval
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: /home/zzc/mydata/mdt/mdtv_1_d/logs/2025-03-26_09-57-27/wandb/wandb/run-20250326_095728-3xxoe903/logs
disconnecting id 0 from server

3.maybe something go wrong but i can't find it,Thank you for your help

[mbreuss]

Looks okay at a first glance. Do you have some rollotu videos that show the behavior? Can you try to load another checkpoint if it has the same problems now?

[ZZC-CN]

Looks okay at a first glance. Do you have some rollotu videos that show the behavior? Can you try to load another checkpoint if it has the same problems now?

Yes,I have rollout videos.

and i have tried to another checkpoint with mdtv_1_abcd in Figure.

Maybe some parameter is wrong like seed,but most of parameter used default parameters,and I get the results like this.

[mbreuss]

Okay, that seems to be something wrong with data loading.
I had a person mentioned an issue with the voltron weights before a month ago. Could you check if the Voltron loading works?
If I have time, I try to have a look myself next week, sorry for the inconvenience!

[ZZC-CN]

Okay, that seems to be something wrong with data loading. I had a person mentioned an issue with the voltron weights before a month ago. Could you check if the Voltron loading works? If I have time, I try to have a look myself next week, sorry for the inconvenience!

Thank you,i will try it by myself,if i have problem i will ask you again

[mbreuss]

Can you try the CNN version? if that still works at least the we know the issue is Voltron

[ZZC-CN]

Can you try the CNN version? if that still works at least the we know the issue is Voltron

Okay,i will try.

[mbreuss]

Did this one work?

[ZZC-CN]

Did this one work?

I haven't tried it yet, but I should have it done by Monday or Tuesday.

[mbreuss]

Okay, feel free to reach out on email.