Forbidden Access to S3 bucket after domain initialization

[salemn]

Describe the bug
I'm not able to deploy a HuggingFaceModel from local development environment due to forbidden access error while calling HeadBucket operation

To reproduce

• Set up single user domain on sagemaker
• Created new user with policy iam:PassRole on execution role generated during domain creation
• Create Access token for the aforementioned user
• Run AWS configure with the access token newly created
• Run the following code

import sagemaker
import boto3
from sagemaker.huggingface.model import HuggingFaceModel

iam_client = boto3.client('iam')
role = iam_client.get_role(RoleName='AmazonSageMaker-ExecutionRole-XXXX')['Role']['Arn']
sess = sagemaker.Session(boto_session=boto3.session.Session(region_name="eu-north-1"))
model_name = 'google/flan-t5-base'
# Hub model configuration <https://huggingface.co/models>
hub = {
    'HF_MODEL_ID': model_name,  # model_id from hf.co/models
    'HF_TASK': 'summarization'  # NLP task you want to use for predictions
}

huggingface_model = HuggingFaceModel(
    env=hub, 
    role=role,  
    transformers_version="4.26", 
    pytorch_version="1.13.1", 
    py_version='py39', 
)

predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.xlarge"
)

Expected behavior
Should deploy model

Screenshots or logs

Bucket sagemaker-eu-north-1-XXXXXXXXXXXX exists, but access is forbidden. Please try again after adding appropriate access.
Traceback (most recent call last):
  File "/Users/naoufel/Work/Pers/model/Sample/main/SM_summary.py", line 22, in <module>
    predictor = huggingface_model.deploy(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/huggingface/model.py", line 319, in deploy
    return super(HuggingFaceModel, self).deploy(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 1695, in deploy
    self._create_sagemaker_model(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 930, in _create_sagemaker_model
    container_def = self.prepare_container_def(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/huggingface/model.py", line 524, in prepare_container_def
    self._upload_code(deploy_key_prefix, repack=True)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/model.py", line 756, in _upload_code
    bucket, key_prefix = s3.determine_bucket_and_prefix(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/s3_utils.py", line 147, in determine_bucket_and_prefix
    final_bucket = sagemaker_session.default_bucket()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 602, in default_bucket
    self._create_s3_bucket_if_it_does_not_exist(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 634, in _create_s3_bucket_if_it_does_not_exist
    self.general_bucket_check_if_user_has_permission(bucket_name, s3, bucket, region, True)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/sagemaker/session.py", line 685, in general_bucket_check_if_user_has_permission
    s3.meta.client.head_bucket(Bucket=bucket_name)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/botocore/client.py", line 1017, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.227.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): 1.13.1
• Framework version:
• Python version: Py39
• CPU or GPU:
• Custom Docker image (Y/N):

Additional context
Add any other context about the problem here.