fs = s3fs.S3FileSystem(profile_name='profile') TypeError: __init__() got an unexpected keyword argument 'profile_name'

[niklahun]

What happened:

Wanted to use s3fs with an AWS profilename.

What you expected to happen:

Connect and work with s3_fs.

Minimal Complete Verifiable Example:

tried:

s3_fs = s3fs.S3FileSystem(profile_name='role')

TypeError: init() got an unexpected keyword argument 'profile_name'

from s3fs import S3FileSystem
import boto3
session = boto3.session.Session(profile_name="aws-biadmin", region_name='us-east-1')
s3_fs = S3FileSystem(anon=False)

NoCredentialsError: Unable to locate credentials

from s3fs.core import S3FileSystem

s3_fs = S3FileSystem(anon=False, client_kwargs={'profile_name':"aws-biadmin", 'region_name': 'us-east-1'})

TypeError: create_client() got an unexpected keyword argument 'profile_name'

from s3fs.core import S3FileSystem

s3_fs = S3FileSystem(anon=False, s3_additional_kwargs={'profile_name':"aws-biadmin", 'region_name': 'us-east-1'})
## This finally accepts it, but does not convey it
s3_fs.ls('s3://bucket/folder')

NoCredentialsError: Unable to locate credentials

from s3fs.core import S3FileSystem
s3_fs = S3FileSystem(anon=False, config_kwargs={'profile_name':"aws-biadmin"})

TypeError: Got unexpected keyword argument 'profile_name

Anything else we need to know?
Multiple past issues states that kwargs should work

Environment:

• s3fs version: '0.4.2'
• Python version: sys.version_info(major=3, minor=7, micro=1, releaselevel='final', serial=0)
• Operating System: MAC OSX
• Install method (conda, pip, source): pip3

[martindurant]

Should have been "profile_name" -> "profile"

[niklahun]

Hi Martin!
Thanks!
It works! I wonder why this does not follow the boto3 naming of this feature thought..

[martindurant]

Perhaps botocore has a different name here? s3fs uses botocore, not boto3, since the last couple of releases, because the former can be swapped for aiobotocore.

[nivcoh]

Hi @martindurant
I just upgrade s3fs from 0.1.5 to 0.5.0 and I can't load any parquet files from S3 using aws profile.
I get the same error as @niklahun for both "profile" and "profile_name"

_create_client() got an unexpected keyword argument 'profile'
any suggestions?

[martindurant]

@nivcoh : could you please try with the following change?

--- a/s3fs/core.py
+++ b/s3fs/core.py
@@ -301,7 +311,7 @@ class S3FileSystem(AsyncFileSystem):
             config_kwargs["signature_version"] = UNSIGNED
         conf = AioConfig(**config_kwargs)
         if self.session is None:
-            self.session = aiobotocore.get_session(**self.kwargs)
+            self.session = aiobotocore.AioSession(**self.kwargs)
         s3creator = self.session.create_client('s3', config=conf, **init_kwargs, **client_kwargs)
         self._s3 = await s3creator.__aenter__()

-edited-

[nivcoh]

Thank you @martindurant & @TomAugspurger but it doesn't seem to work. (I tried using both "profile" / "profile_name")

here is the full print error:
##################################################################

~\AppData\Local\Programs\Python\Python37\lib\site-packages\fsspec\spec.py in call(cls, *args, **kwargs)
53 return cls._cache[token]
54 else:
---> 55 obj = super().call(*args, **kwargs)
56 # Setting _fs_token here causes some static linters to complain.
57 obj.fs_token = token

~\AppData\Local\Programs\Python\Python37\lib\site-packages\s3fs\core.py in init(self, anon, key, secret, token, use_ssl, client_kwargs, requester_pays, default_block_size, default_fill_cache, default_cache_type, version_aware, config_kwargs, s3_additional_kwargs, session, username, password, asynchronous, loop, **kwargs)
183 self.use_ssl = use_ssl
184 if not asynchronous:
--> 185 self.connect()
186 weakref.finalize(self, sync, self.loop, self._s3.close)
187 else:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\fsspec\asyn.py in wrapper(*args, **kwargs)
98 def wrapper(*args, **kwargs):
99 self = obj or args[0]
--> 100 return maybe_sync(func, self, *args, **kwargs)
101
102 return wrapper

~\AppData\Local\Programs\Python\Python37\lib\site-packages\fsspec\asyn.py in maybe_sync(func, self, *args, **kwargs)
78 if inspect.iscoroutinefunction(func):
79 # run the awaitable on the loop
---> 80 return sync(loop, func, *args, **kwargs)
81 else:
82 # just call the blocking function

~\AppData\Local\Programs\Python\Python37\lib\site-packages\fsspec\asyn.py in sync(loop, func, callback_timeout, *args, **kwargs)
49 if error[0]:
50 typ, exc, tb = error[0]
---> 51 raise exc.with_traceback(tb)
52 else:
53 return result[0]

~\AppData\Local\Programs\Python\Python37\lib\site-packages\fsspec\asyn.py in f()
33 if callback_timeout is not None:
34 future = asyncio.wait_for(future, callback_timeout)
---> 35 result[0] = await future
36 except Exception:
37 error[0] = sys.exc_info()

~\AppData\Local\Programs\Python\Python37\lib\site-packages\s3fs\core.py in _connect(self, kwargs)
303 if self.session is None:
304 self.session = aiobotocore.AioSession(**self.kwargs)
--> 305 s3creator = self.session.create_client('s3', config=conf, **init_kwargs, **client_kwargs)
306 self._s3 = await s3creator.aenter()
307 self._kwargs_helper = ParamKwargsHelper(self.s3)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\aiobotocore\session.py in create_client(self, *args, **kwargs)
46
47 def create_client(self, *args, **kwargs):
---> 48 return ClientCreatorContext(self._create_client(*args, **kwargs))
49
50 async def _create_client(self, service_name, region_name=None,

TypeError: _create_client() got an unexpected keyword argument 'profile'

[martindurant]

Dod you call as s3fs.S3FileSystem(profile='role') or something else?

[nivcoh]

s3_fs = S3FileSystem(anon=False, client_kwargs={'profile_name':"prod"})
s3_fs.ls('s3://myBucket/')

[nivcoh]

changing to
s3_fs = s3fs.S3FileSystem(profile='prod')
works 👍

[martindurant]

OK!

[nivcoh]

when will you expect this fix to be released?

[martindurant]

#351 is adding this, but there are ongoing problems with aiobotocore interaction for regional buckets. I would like to see those fixed for a bugfix release, so hoping ~days.

[nivcoh]

this will be great :) Thank you for your hard work & kind support