SAM decoder training error

[alexdzm]

System Info

• transformers version: 4.55.2
• Platform: Linux-4.18.0-553.16.1.el8_10.cloud.0.1.x86_64-x86_64-with-glibc2.28
• Python version: 3.12.10
• Huggingface_hub version: 0.34.4
• Safetensors version: 0.6.2
• Accelerate version: 1.10.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: Yes
• GPU type: NVIDIA A100-SXM4-80GB

Who can help?

@amyeroberts, @qubvel, @vasqu

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Training SAM with a transformers trainer causes this error. It seems that dropout_p is not defined anywhere for this object and this logic was updated in a recent PR. When I manually replace this if else with a set dropout value it works. It would be nice to remove this please!

This can be reproduced by running the sam_trainer.py script in this repo

Exception has occurred: AttributeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
'SamAttention' object has no attribute 'dropout_p'
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1962, in __getattr__
    raise AttributeError(
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/models/sam/modeling_sam.py", line 269, in forward
    dropout=0.0 if not self.training else self.dropout_p,
                                          ^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/models/sam/modeling_sam.py", line 325, in forward
    queries, _ = self.self_attn(query=queries, key=queries, value=queries)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/models/sam/modeling_sam.py", line 397, in forward
    queries, keys, _ = layer(
                       ^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/models/sam/modeling_sam.py", line 512, in forward
    point_embedding, image_embeddings = self.transformer(
                                        ^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/models/sam/modeling_sam.py", line 1353, in forward
    low_res_masks, iou_predictions = self.mask_decoder(
                                     ^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1083, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/accelerate/utils/operations.py", line 806, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/accelerate/utils/operations.py", line 818, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/sam_trainer.py", line 119, in compute_loss
    outputs = model(**inputs, multimask_output=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 3796, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2582, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/.venv/lib/python3.12/site-packages/transformers/trainer.py", line 2238, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/sam_trainer.py", line 234, in main
    train_result = trainer.train()
                   ^^^^^^^^^^^^^^^
  File "/mnt/data1/extract-finetuning/sam_trainer.py", line 240, in <module>
    main()
  File "/mnt/data1/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/mnt/data1/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/runpy.py", line 198, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'SamAttention' object has no attribute 'dropout_p'

Expected behavior

The model should train without throwing this error

[vasqu]

That's a good catch @alexdzm, looks like it was unintentionally removed in the init (see

transformers/src/transformers/models/sam/modeling_sam.py

Line 740 in c6ee0b1

self.dropout = config.attention_dropout

)
Can you readd this (as dropout_p instead) and check. cc @yonigozlan for a sanity check :D

Would you open a PR for that fix? It would be that fix + a small test to make sure we don't encounter this issue again.

[alexdzm]

Yes I'll write a fix for this. Thanks!

[yonigozlan]

Thanks for catching this @alexdzm, can you open a PR to fix it? Otherwise I'll open one :)

[alexdzm]

Hey @yonigozlan I am unlikely to get to this for the next two weeks. Please crack on and link your PR here!