Skip to content

combination of managed layer AWSSDKPandas-Python312 and AWSLambdaPowertoolsPythonV2 result in ModuleNotFoundError: No module named '_cffi_backend' #2688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Eerk1975 opened this issue Feb 24, 2024 · 6 comments
Labels
bug Something isn't working

Comments

@Eerk1975
Copy link

Describe the bug

For my Python 3.12 Lambda function I need to combine Powertools and SDK for Pandas. I would like to use the managed layers for this.
I added layers: arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python312:4 and arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:64

this results in:

Function Logs
rom cryptography.hazmat.primitives.hashes import HashAlgorithm
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 4103.15 ms Phase: init Status: error Error Type: Runtime.ExitError
ModuleNotFoundError: No module named '_cffi_backend'
thread '' panicked at /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/err/mod.rs:788:5:
Python API call failed
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Traceback (most recent call last):
File "/var/runtime/bootstrap.py", line 63, in
main()
File "/var/runtime/bootstrap.py", line 60, in main
awslambdaricmain.main([os.environ["LAMBDA_TASK_ROOT"], os.environ["_HANDLER"]])
File "/var/lang/lib/python3.12/site-packages/awslambdaric/main.py", line 21, in main
bootstrap.run(app_root, handler, lambda_runtime_api_addr)
File "/var/lang/lib/python3.12/site-packages/awslambdaric/bootstrap.py", line 478, in run
request_handler = _get_handler(handler)
^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/site-packages/awslambdaric/bootstrap.py", line 53, in _get_handler
m = importlib.import_module(modname.replace("/", "."))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 994, in exec_module
File "", line 488, in _call_with_frames_removed
File "/var/task/lambda_function.py", line 2, in
import awswrangler as wr
File "/opt/python/awswrangler/init.py", line 10, in
from awswrangler import (
File "/opt/python/awswrangler/mysql.py", line 27, in
pymysql = _utils.import_optional_dependency("pymysql")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/awswrangler/_utils.py", line 147, in import_optional_dependency
module = importlib.import_module(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/pymysql/init.py", line 78, in
from . import connections # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/pymysql/connections.py", line 13, in
from . import _auth
File "/opt/python/pymysql/_auth.py", line 9, in
from cryptography.hazmat.primitives import serialization, hashes
File "/opt/python/cryptography/hazmat/primitives/serialization/init.py", line 7, in
from cryptography.hazmat.primitives._serialization import (
File "/opt/python/cryptography/hazmat/primitives/_serialization.py", line 10, in
from cryptography.hazmat.primitives.hashes import HashAlgorithm
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 58089.99 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: 5d231810-7e77-403c-8727-30dba414f36f Version: $LATEST
RequestId: 5d231810-7e77-403c-8727-30dba414f36f Error: Runtime exited with error: exit status 1

I also tried the same with Python 3.11 and corresponding managed layers, but tis resulted in the same behaviour.
removing one of the layers resulted in behaviour I expected.

How to Reproduce

as code in the lambda function I only added (as test):

import awswrangler as wr

def lambda_handler(event, context):

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }``` 



### Expected behavior

Hello from Lambda!'

### Your project

_No response_

### Screenshots

_No response_

### OS

Lambda

### Python version

3.12

### AWS SDK for pandas version

3.6.0

### Additional context

_No response_
@Eerk1975 Eerk1975 added the bug Something isn't working label Feb 24, 2024
@Eerk1975
Copy link
Author

an extra test I did: the combination of arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python311:6 and arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:11 in Python 3.11 did work

@jaidisido
Copy link
Contributor

jaidisido commented Feb 26, 2024

So here's what is going on here. awswrangler packages pymysql as a dependency in its lambda layers. In turn, pymysql has an optional dependency on the cryptography package. Because it's optional, we don't package it in the zip layer and therefore pymysql does not try to import it. This explains why you have no issue when using the awswrangler layer on its own.

On the other hand, recent versions of aws_lambda_powertools do package cryptography as a dependency in their layer. As a result, when both layers are used, pymysql detects that cryptography is available from powertools and it tries to import it.

Now on the exception No module named '_cffi_backend'. There seems to be an issue with the way cryptography was packaged in the powertools layer. I am able to reproduce the exception with this simple code:

import json

from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
    AWSEncryptionSDKProvider,
)

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

which also raises No module named _cffi_backend and as you can see does not involve awswrangler. From the little research I have done, it looks like powertools uses the same base image to package their layers but cffi is sensitive to the base python version on which it's installed:
https://stackoverflow.com/questions/76443834/aws-lambda-python-3-10-no-module-named-cffi-backend
We have the same issue with numpy and this is why we have one layer per python version when powertools does not.

In short, this is a powertools specific issue that they need to solve. I can open an issue with them to inform them.

@heitorlessa
Copy link

Powertools for AWS Lambda on-call checking in - we're investigating. Thank you so much for the preliminary investigation.

@heitorlessa
Copy link

heitorlessa commented Feb 26, 2024

we're making a patch release now to remove aws-encryption-sdk from the Powertools Layer. I'll update as soon as it's out -- it'll be version 65.

@jaidisido
Copy link
Contributor

After testing with version 2.34.2 of powertools using layers:

  • arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:65
  • arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python312:4

on Python 3.12, I can confirm that the issue is fixed, thanks @heitorlessa. Closing the ticket

@heitorlessa
Copy link

an extra note for whoever bumps into this -- we will be investigating the operational effort in adding a new layer per Python version in the near future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants