Exception on import when AWS_PROFILE is invalid/null

[lorengordon]

Expected Behavior

import aws_lambda_powertools should still succeed even if the ENV contains AWS_PROFILE with a bad value.

Current Behavior

It appears that during import, under at least some conditions, aws_lambda_powertools will attempt to setup an aws client session? I'm not sure I understand why. So if the ENV contains an invalid AWS config, in particular AWS_PROFILE= or AWS_PROFILE=fakeprofile, then the import fails with an exception. Which, yes, it's invalid, and I now that I know aws_lambda_powertools has this behavior I can work around it easily by unsetting the envs. But it still seems like it shouldn't be setting up a session on import?

Possible Solution

Not sure on a solution, but a workaround is to make sure your development environment unsets bad AWS_ envs before attempting to run tests on your python code. :)

Steps to Reproduce (for bugs)

1. AWS_PROFILE= python -c 'import aws_lambda_powertools'

Environment

• Powertools version used: 1.10.5
• Packaging format (Layers, PyPi): PyPi
• AWS Lambda function runtime:: n/a
• Debugging logs

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/__init__.py", line 9, in <module>
    from .tracing import Tracer  # noqa: F401
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/tracing/__init__.py", line 6, in <module>
    from .tracer import Tracer
  File "/home/loren/.local/lib/python3.6/site-packages/aws_lambda_powertools/tracing/tracer.py", line 10, in <module>
    import aws_xray_sdk.core
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/__init__.py", line 10, in <module>
    xray_recorder = AsyncAWSXRayRecorder()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 66, in __init__
    self._sampler = DefaultSampler()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/sampler.py", line 25, in __init__
    self._connector = ServiceConnector()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 23, in __init__
    self._xray_client = self._create_xray_client()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 164, in _create_xray_client
    aws_access_key_id='', aws_secret_access_key=''
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 802, in create_client
    verify = self.get_config_variable('ca_bundle')
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 241, in get_config_variable
    logical_name)
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 340, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile () could not be found

[michaelbrewer]

@lorengordon this has to do with importing the xray core, which is then picking up the AWS_PROFILE being set.

I would either unset AWS_PROFILE and have dummy values for the rest

unset AWS_PROFILE 
export AWS_ACCESS_KEY_ID=fake
export AWS_SECRET_ACCESS_KEY=fake
export AWS_DEFAULT_REGION=us-east-1

Or unset it before importing powertools in the tests

import os

if "AWS_PROFILE" in os.environ:
    del os.environ["AWS_PROFILE"]

[lorengordon]

@michaelbrewer Of course, that's my workaround. When I first attempted to reproduce, I imported the xray core directly and it worked fine, so I thought it was something in this library. But I must have messed up the env, because now I am reproducing easily on that import. I'll open an issue on the xray sdk. Thanks!

$ env |grep AWS_
AWS_PROFILE=

$ python -c 'import aws_xray_sdk.core'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/__init__.py", line 10, in <module>
    xray_recorder = AsyncAWSXRayRecorder()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 66, in __init__
    self._sampler = DefaultSampler()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/sampler.py", line 25, in __init__
    self._connector = ServiceConnector()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 23, in __init__
    self._xray_client = self._create_xray_client()
  File "/home/loren/.local/lib/python3.6/site-packages/aws_xray_sdk/core/sampling/connector.py", line 164, in _create_xray_client
    aws_access_key_id='', aws_secret_access_key=''
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 802, in create_client
    verify = self.get_config_variable('ca_bundle')
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 241, in get_config_variable
    logical_name)
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 313, in get_config_variable
    return provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 410, in provide
    value = provider.provide()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/configprovider.py", line 471, in provide
    scoped_config = self._session.get_scoped_config()
  File "/home/loren/.local/lib/python3.6/site-packages/botocore/session.py", line 340, in get_scoped_config
    raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile () could not be found

[lorengordon]

I suppose it could be a feature request here, in the sense that I don't need or want xray right now. I am only using the logger. So it might be nice to gate the import to an explicit call to a powertools class/method...

[michaelbrewer]

@heitorlessa is looking at lazy loading libraries

[heitorlessa]

100% and thanks @michaelbrewer for helping out.

I'm not sure how I missed this issue in my notifications so apologies @lorengordon for not replying earlier.

I've done some research on backwards compatible solutions that work for 3.6+ (what we support), and am scheduling next Friday to experiment and will keep you posted.

Thanks for reporting it!

[lorengordon]

@heitorlessa All good, the workarounds are viable for now. 3.6+ works for us. Feel free to close this if you have another tracking issue for lazy loading libraries... I opened another issue on the xray sdk to better understand why it needs to setup the session as a global, and impact imports like this...

[heitorlessa]

Awesome - I'll subscribe to that.

I'll keep this open, and I'll try this method first as it seems clever enough to also prevent PyCharm/VSCode debugger to crash due to increased ref counting in getattr

https://wil.yegelwel.com/lazily-importing-python-modules/

[lorengordon]

Nice, and it's very simple! I've got use cases for that in my own projects! 😂

[michaelbrewer]

As a pure hack, i have disabled the one i don't need during my unit testing

import sys
from unittest.mock import Mock

mock = Mock()
sys.modules['aws_lambda_powertools.tracing']=mock
sys.modules['aws_lambda_powertools.metrics']=mock
sys.modules['aws_lambda_powertools.loggin']=mock