.NET AWS Lambda function using .NET 7 RC1 fails to start when using a custom runtime on AWS Linux 2 on arm64

[martincostello]

Description

This issue is a duplicate of aws/aws-lambda-dotnet#1310 that I'm also opening for visibility in case some collaboration is needed between Microsoft and AWS to resolve this issue to enable .NET 7 on AWS Lambda.

I attempted to deploy a .NET Lambda function I have updated from .NET 6 to .NET 7 to AWS Lambda using a custom runtime (deployment).

However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in AWS CloudWatch show that the function is failing to start.

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.

I'm not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I'm all ears to unblock early adoption.

/cc @normj

Reproduction Steps

An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.

Expected behavior

The .NET Lambda function starts successfully.

Actual behavior

The Lambda function fails to start with the following error:

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

Regression?

To the end-user, yes, but the change in the GLIBC version is an intentional change.

Known Workarounds

None.

Configuration

• .NET SDK: 7.0.100-rc.1.22431.12
• Operating system: Amazon Linux 2 on Graviton/arm64 (Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022)

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

.NET on arm64 requires a Linux distro with glibc 2.27 or newer. It roughly matches to Ubuntu 18.04 or newer.

This is documented at https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md

[jkotas]

https://repost.aws/questions/QUrXOioL46RcCnFGyELJWKLw/glibc-2-27-on-amazon-linux-2

[normj]

@jkotas Can you confirm that 2.27 is only required for ARM and not X64? Out of curiosity what was the feature in 2.27 that was needed for .NET 7 as this is potentially going to block a fair amount of use cases on AWS.

[martincostello]

@normj Some digging was done by @slang25 earlier when we found this issue with Lambda, and this appears to be the main motivation: 31cd7eb

[jkotas]

The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.

CentOS 7 that we use to build for x64 does not work well on Arm64. And Ubuntu 16 that we use to build for arm64 in .NET 6 is out of mainstream support.

The change you have linked was just a follow up cleanup.

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

This issue is a duplicate of aws/aws-lambda-dotnet#1310 that I'm also opening for visibility in case some collaboration is needed between Microsoft and AWS to resolve this issue to enable .NET 7 on AWS Lambda.

I attempted to deploy a .NET Lambda function I have updated from .NET 6 to .NET 7 to AWS Lambda using a custom runtime (deployment).

However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in AWS CloudWatch show that the function is failing to start.

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.

I'm not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I'm all ears to unblock early adoption.

/cc @normj

Reproduction Steps

An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.

Expected behavior

The .NET Lambda function starts successfully.

Actual behavior

The Lambda function fails to start with the following error:

Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)

Regression?

To the end-user, yes, but the change in the GLIBC version is an intentional change.

Known Workarounds

None.

Configuration

• .NET SDK: 7.0.100-rc.1.22431.12
• Operating system: Amazon Linux 2 on Graviton/arm64 (Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022)

Other information

No response

Author:	martincostello
Assignees:	-
Labels:	question, area-Meta
Milestone:	-

[jeffschwMSFT]

closing this issue as answered, but we can reopen if necessary.

[sebpop]

@jkotas wrote:

The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.
CentOS 7 that we use to build for x64 does not work well on Arm64. And Ubuntu 16 that we use to build for arm64 in .NET 6 is out of mainstream support.

Amazon Linux 2 is based on CentOS with a glibc older than Ubuntu 18.04:

[al2] $ /lib64/libc.so.6 
GNU C Library (GNU libc) stable release version 2.26, by Roland McGrath et al.

[ubuntu 18.04] $ /lib/x86_64-linux-gnu/libc.so.6 
GNU C Library (Ubuntu GLIBC 2.27-3ubuntu1.6) stable release version 2.27.

Could the .net framework be built for arm64 on AL2 that has the oldest version of glibc?
AL2 is supported on ARM64 and will be supported until at least 2024-06-30.

From https://aws.amazon.com/amazon-linux-2/faqs/

Amazon Linux 2 end of support date (End of Life, or EOL) has been extended by one year from 2023-06-30 to 2024-06-30

Thank you,
Sebastian

[AdamTylerLynch]

From the supported OS versions page, I note that you compile on 2.17 for x64 (CentOS7), but 2.27 on Ubuntu.

If you could compile on 2.26 that will fix the issue.

• x64: glibc 2.17 (from CentOS 7)
• Arm32, Arm64: glibc 2.27 (from Ubuntu 18.04)
• Alpine (x64 and Arm64): musl 1.2.2 (from Alpine 3.15)