"sam build -u -e TARGET_CC=foo" leads to "cargo: Command not found"

[brainstorm]

Description:

I'm trying to pass build variables to the sam build docker container to solve briansmith/ring#1332 cross-compiling issue, also pointed out over twitter.

Steps to reproduce:

On an Apple Silicon M1 laptop, run the following:

brew tap aws/tap
brew install aws-sam-cli
git clone https://github.com/umccr/s3-rust-noodles-bam && cd s3-rust-noodles-bam && git checkout s3-server
sam build -u -e FOO=BAR

Observed result:

When sam build is called with environment variable container-passing flag, cargo command is not found:

(base) rvalls@m1 s3-rust-noodles-bam % sam build --debug --use-container -e TARGET_CC=gcc
2021-08-03 12:46:09,833 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-08-03 12:46:09,834 | Using config file: samconfig.toml, config environment: default
2021-08-03 12:46:09,834 | Expand command line arguments to:
2021-08-03 12:46:09,834 | --template_file=/Users/rvalls/dev/umccr/s3-rust-noodles-bam/template.yml --use_container --container_env_var=('TARGET_CC=gcc',) --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2021-08-03 12:46:09,887 | 'build' command is called
2021-08-03 12:46:09,887 | Starting Build inside a container
2021-08-03 12:46:09,889 | No Parameters detected in the template
2021-08-03 12:46:09,901 | 2 stacks found in the template
2021-08-03 12:46:09,901 | No Parameters detected in the template
2021-08-03 12:46:09,916 | 2 resources found in the stack
2021-08-03 12:46:09,916 | No Parameters detected in the template
2021-08-03 12:46:09,925 | Found Serverless function with name='s3Bam' and CodeUri='.'
2021-08-03 12:46:09,925 | --base-dir is not presented, adjusting uri . relative to /Users/rvalls/dev/umccr/s3-rust-noodles-bam/template.yml
2021-08-03 12:46:09,925 | No Parameters detected in the template
2021-08-03 12:46:09,935 | Instantiating build definitions
2021-08-03 12:46:09,937 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(provided.al2, /Users/rvalls/dev/umccr/s3-rust-noodles-bam, Zip, , 6cd8efd1-34b8-4609-9a74-ac919577cfcd, {'BuildMethod': 'makefile'}, {'TARGET_CC': 'gcc'}, []), Function: Function(name='s3Bam', functionname='s3-bamheader', runtime='provided.al2', memory=128, timeout=30, handler='bootstrap.is.real.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/rvalls/dev/umccr/s3-rust-noodles-bam', environment=None, rolearn=None, layers=[], events={'RootEvent': {'Type': 'Api', 'Properties': {'Path': '/', 'Method': 'any', 'RestApiId': 'ServerlessRestApi'}}, 'GatewayEvent': {'Type': 'Api', 'Properties': {'Path': '/{proxy+}', 'Method': 'any', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'makefile'}, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-08-03 12:46:09,938 | Building codeuri: /Users/rvalls/dev/umccr/s3-rust-noodles-bam runtime: provided.al2 metadata: {'BuildMethod': 'makefile'} functions: ['s3Bam']
2021-08-03 12:46:09,938 | Building to following folder /Users/rvalls/dev/umccr/s3-rust-noodles-bam/.aws-sam/build/s3Bam

Fetching public.ecr.aws/sam/build-provided.al2:latest Docker container image......
2021-08-03 12:46:13,841 | Mounting /Users/rvalls/dev/umccr/s3-rust-noodles-bam as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Running workflow 'CustomMakeBuilder'
Running CustomMakeBuilder:CopySource
CustomMakeBuilder:CopySource succeeded
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
executing Make: ['make', '--makefile', '/tmp/samcli/source/Makefile', 'build-s3Bam']
CustomMakeBuilder:MakeBuild failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 94, in execute
    cwd=self.scratch_dir,
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/make.py", line 88, in run
    raise MakeExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error 127

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 97, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error 127
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 94, in execute
    cwd=self.scratch_dir,
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/make.py", line 88, in run
    raise MakeExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error 127

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 97, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error 127

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/__main__.py", line 126, in main
    mode=params.get("mode", None),
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/builder.py", line 125, in build
    return workflow.run()
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 76, in wrapper
    func(self, *args, **kwargs)
  File "/usr/local/opt/sam-cli/lib64/python3.7/site-packages/aws_lambda_builders/workflow.py", line 276, in run
    raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: CustomMakeBuilder:MakeBuild - Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error 127
2021-08-03 12:47:53,527 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "CustomMakeBuilder:MakeBuild - Make Failed: make: cargo: Command not found\nmake: *** [build-s3Bam] Error 127"}}

Build Failed
2021-08-03 12:47:54,752 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '09ec9ada-8b55-412e-8304-b82ef7beff57', 'installationId': 'b0a4feb6-813d-4f61-952f-5919f29d6b17', 'sessionId': '684afe41-d857-4475-8ce5-40368e98041e', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.27.2', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 104915, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2021-08-03 12:47:55,650 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: CustomMakeBuilder:MakeBuild - Make Failed: make: cargo: Command not found
make: *** [build-s3Bam] Error **127**

When calling sam build with no arguments, cargo seems to be there in the provided.al2 container:

(base) rvalls@m1 s3-rust-noodles-bam % sam build --debug
2021-08-03 12:50:43,892 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-08-03 12:50:43,892 | Using config file: samconfig.toml, config environment: default
2021-08-03 12:50:43,892 | Expand command line arguments to:
2021-08-03 12:50:43,892 | --template_file=/Users/rvalls/dev/umccr/s3-rust-noodles-bam/template.yml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache
2021-08-03 12:50:43,940 | 'build' command is called
2021-08-03 12:50:43,942 | No Parameters detected in the template
2021-08-03 12:50:43,953 | 2 stacks found in the template
2021-08-03 12:50:43,953 | No Parameters detected in the template
2021-08-03 12:50:43,968 | 2 resources found in the stack
2021-08-03 12:50:43,968 | No Parameters detected in the template
2021-08-03 12:50:43,977 | Found Serverless function with name='s3Bam' and CodeUri='.'
2021-08-03 12:50:43,978 | --base-dir is not presented, adjusting uri . relative to /Users/rvalls/dev/umccr/s3-rust-noodles-bam/template.yml
2021-08-03 12:50:43,978 | No Parameters detected in the template
2021-08-03 12:50:43,987 | Instantiating build definitions
2021-08-03 12:50:43,989 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(provided.al2, /Users/rvalls/dev/umccr/s3-rust-noodles-bam, Zip, , 32329348-bbde-4e05-be4c-45e7c4718426, {'BuildMethod': 'makefile'}, {}, []), Function: Function(name='s3Bam', functionname='s3-bamheader', runtime='provided.al2', memory=128, timeout=30, handler='bootstrap.is.real.handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/rvalls/dev/umccr/s3-rust-noodles-bam', environment=None, rolearn=None, layers=[], events={'RootEvent': {'Type': 'Api', 'Properties': {'Path': '/', 'Method': 'any', 'RestApiId': 'ServerlessRestApi'}}, 'GatewayEvent': {'Type': 'Api', 'Properties': {'Path': '/{proxy+}', 'Method': 'any', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'makefile'}, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-08-03 12:50:43,990 | Building codeuri: /Users/rvalls/dev/umccr/s3-rust-noodles-bam runtime: provided.al2 metadata: {'BuildMethod': 'makefile'} functions: ['s3Bam']
2021-08-03 12:50:43,990 | Building to following folder /Users/rvalls/dev/umccr/s3-rust-noodles-bam/.aws-sam/build/s3Bam
2021-08-03 12:50:43,991 | Loading workflow module 'aws_lambda_builders.workflows'
2021-08-03 12:50:43,993 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-08-03 12:50:43,994 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-08-03 12:50:43,995 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-08-03 12:50:43,996 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-08-03 12:50:43,997 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-08-03 12:50:43,998 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-08-03 12:50:43,999 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-08-03 12:50:44,000 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-08-03 12:50:44,001 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-08-03 12:50:44,001 | Found workflow 'CustomMakeBuilder' to support capabilities 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-08-03 12:50:44,001 | Running workflow 'CustomMakeBuilder'
2021-08-03 12:50:44,001 | Running CustomMakeBuilder:CopySource
2021-08-03 12:50:48,306 | CustomMakeBuilder:CopySource succeeded
2021-08-03 12:50:48,306 | Running CustomMakeBuilder:MakeBuild
2021-08-03 12:50:48,306 | Current Artifacts Directory : /Users/rvalls/dev/umccr/s3-rust-noodles-bam/.aws-sam/build/s3Bam
2021-08-03 12:50:48,306 | executing Make: ['make', '--makefile', '/Users/rvalls/dev/umccr/s3-rust-noodles-bam/Makefile', 'build-s3Bam']
2021-08-03 12:50:56,801 | CustomMakeBuilder:MakeBuild failed
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.27.2/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 87, in execute
    self.subprocess_make.run(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.27.2/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/make.py", line 88, in run
    raise MakeExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.27
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.73
   Compiling libc v0.2.97
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.0
   Compiling log v0.4.14
   Compiling pin-project-lite v0.2.7
   Compiling futures-core v0.3.15
   Compiling bytes v1.0.1
   Compiling itoa v0.4.7
   Compiling lazy_static v1.4.0
   Compiling proc-macro-hack v0.5.19
   Compiling cc v1.0.68
   Compiling futures-sink v0.3.15
   Compiling futures-task v0.3.15
   Compiling futures-channel v0.3.15
   Compiling proc-macro-nested v0.1.7
   Compiling futures-io v0.3.15
   Compiling pin-utils v0.1.0
   Compiling slab v0.4.3
   Compiling fnv v1.0.7
   Compiling hashbrown v0.11.2
   Compiling once_cell v1.8.0
   Compiling serde_derive v1.0.126
   Compiling serde v1.0.126
   Compiling ryu v1.0.5
   Compiling httparse v1.4.1
   Compiling percent-encoding v2.1.0
   Compiling spin v0.5.2
   Compiling untrusted v0.7.1
   Compiling try-lock v0.2.3
   Compiling tower-service v0.3.1
   Compiling version_check v0.9.3
   Compiling ucd-trie v0.1.3
   Compiling httpdate v1.0.1
   Compiling regex-syntax v0.6.25
   Compiling either v1.6.1
   Compiling serde_json v1.0.64
   Compiling matches v0.1.8
   Compiling ansi_term v0.12.1
   Compiling base64 v0.13.0
   Compiling crc32fast v1.2.1
   Compiling xmlparser v0.13.3
   Compiling adler v1.0.2
   Compiling openssl-probe v0.1.4
   Compiling tower-layer v0.3.1
   Compiling diff v0.1.12
   Compiling bitflags v1.2.1
   Compiling hex v0.4.3
   Compiling zeroize v1.3.0
   Compiling tinyvec_macros v0.1.0
   Compiling byteorder v1.4.3
   Compiling fs_extra v1.2.0
   Compiling smallvec v1.6.1
   Compiling fastrand v1.4.1
   Compiling bit-vec v0.6.3
   Compiling md5 v0.7.0
   Compiling futures-macro v0.3.15
   Compiling tokio v1.7.1
   Compiling num-traits v0.2.14
   Compiling indexmap v1.7.0
   Compiling futures-util v0.3.15
   Compiling num-integer v0.1.44
   Compiling miniz_oxide v0.4.4
   Compiling tracing-core v0.1.18
   Compiling sharded-slab v0.1.1
   Compiling http v0.2.4
   Compiling thread_local v1.1.3
   Compiling ring v0.16.20
   Compiling pest v2.1.3
   Compiling bytes-utils v0.1.1
   Compiling form_urlencoded v1.0.1
   Compiling unicode-bidi v0.3.5
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling roxmltree v0.14.1
   Compiling pretty_assertions v0.7.2
   Compiling tinyvec v1.2.0
   Compiling jemalloc-sys v0.3.2
   Compiling regex-automata v0.1.10
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/release/build/ring-0c3d64667514526d/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("aarch64-apple-darwin")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/x86_64-unknown-linux-gnu/release/build/ring-1b5ef9a99e86a98a/out/aesni-x86_64-elf.o" "/Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S:1181:19: error: unexpected token in '.section' directive
  .section .note.GNU-stack,"",@progbits
                    ^
  thread 'main' panicked at 'execution failed', /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [build-s3Bam] Error 101

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.27.2/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.27.2/libexec/lib/python3.8/site-packages/aws_lambda_builders/workflows/custom_make/actions.py", line 97, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: Make Failed: Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.27
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.73
   Compiling libc v0.2.97
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.0
   Compiling log v0.4.14
   Compiling pin-project-lite v0.2.7
   Compiling futures-core v0.3.15
   Compiling bytes v1.0.1
   Compiling itoa v0.4.7
   Compiling lazy_static v1.4.0
   Compiling proc-macro-hack v0.5.19
   Compiling cc v1.0.68
   Compiling futures-sink v0.3.15
   Compiling futures-task v0.3.15
   Compiling futures-channel v0.3.15
   Compiling proc-macro-nested v0.1.7
   Compiling futures-io v0.3.15
   Compiling pin-utils v0.1.0
   Compiling slab v0.4.3
   Compiling fnv v1.0.7
   Compiling hashbrown v0.11.2
   Compiling once_cell v1.8.0
   Compiling serde_derive v1.0.126
   Compiling serde v1.0.126
   Compiling ryu v1.0.5
   Compiling httparse v1.4.1
   Compiling percent-encoding v2.1.0
   Compiling spin v0.5.2
   Compiling untrusted v0.7.1
   Compiling try-lock v0.2.3
   Compiling tower-service v0.3.1
   Compiling version_check v0.9.3
   Compiling ucd-trie v0.1.3
   Compiling httpdate v1.0.1
   Compiling regex-syntax v0.6.25
   Compiling either v1.6.1
   Compiling serde_json v1.0.64
   Compiling matches v0.1.8
   Compiling ansi_term v0.12.1
   Compiling base64 v0.13.0
   Compiling crc32fast v1.2.1
   Compiling xmlparser v0.13.3
   Compiling adler v1.0.2
   Compiling openssl-probe v0.1.4
   Compiling tower-layer v0.3.1
   Compiling diff v0.1.12
   Compiling bitflags v1.2.1
   Compiling hex v0.4.3
   Compiling zeroize v1.3.0
   Compiling tinyvec_macros v0.1.0
   Compiling byteorder v1.4.3
   Compiling fs_extra v1.2.0
   Compiling smallvec v1.6.1
   Compiling fastrand v1.4.1
   Compiling bit-vec v0.6.3
   Compiling md5 v0.7.0
   Compiling futures-macro v0.3.15
   Compiling tokio v1.7.1
   Compiling num-traits v0.2.14
   Compiling indexmap v1.7.0
   Compiling futures-util v0.3.15
   Compiling num-integer v0.1.44
   Compiling miniz_oxide v0.4.4
   Compiling tracing-core v0.1.18
   Compiling sharded-slab v0.1.1
   Compiling http v0.2.4
   Compiling thread_local v1.1.3
   Compiling ring v0.16.20
   Compiling pest v2.1.3
   Compiling bytes-utils v0.1.1
   Compiling form_urlencoded v1.0.1
   Compiling unicode-bidi v0.3.5
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling roxmltree v0.14.1
   Compiling pretty_assertions v0.7.2
   Compiling tinyvec v1.2.0
   Compiling jemalloc-sys v0.3.2
   Compiling regex-automata v0.1.10
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/release/build/ring-0c3d64667514526d/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("aarch64-apple-darwin")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/x86_64-unknown-linux-gnu/release/build/ring-1b5ef9a99e86a98a/out/aesni-x86_64-elf.o" "/Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S:1181:19: error: unexpected token in '.section' directive
  .section .note.GNU-stack,"",@progbits
                    ^
  thread 'main' panicked at 'execution failed', /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [build-s3Bam] Error 101

Build Failed
2021-08-03 12:50:57,258 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'a2e56033-37e4-4bc2-8c39-0a9ec41dba7f', 'installationId': 'b0a4feb6-813d-4f61-952f-5919f29d6b17', 'sessionId': 'f7ac1940-5452-4195-8558-7c51c453afc4', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.11', 'samcliVersion': '1.27.2', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 13364, 'exitReason': 'WorkflowFailedError', 'exitCode': 1}}]}
2021-08-03 12:50:58,193 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: CustomMakeBuilder:MakeBuild - Make Failed: Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.27
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.73
   Compiling libc v0.2.97
   Compiling cfg-if v1.0.0
   Compiling memchr v2.4.0
   Compiling log v0.4.14
   Compiling pin-project-lite v0.2.7
   Compiling futures-core v0.3.15
   Compiling bytes v1.0.1
   Compiling itoa v0.4.7
   Compiling lazy_static v1.4.0
   Compiling proc-macro-hack v0.5.19
   Compiling cc v1.0.68
   Compiling futures-sink v0.3.15
   Compiling futures-task v0.3.15
   Compiling futures-channel v0.3.15
   Compiling proc-macro-nested v0.1.7
   Compiling futures-io v0.3.15
   Compiling pin-utils v0.1.0
   Compiling slab v0.4.3
   Compiling fnv v1.0.7
   Compiling hashbrown v0.11.2
   Compiling once_cell v1.8.0
   Compiling serde_derive v1.0.126
   Compiling serde v1.0.126
   Compiling ryu v1.0.5
   Compiling httparse v1.4.1
   Compiling percent-encoding v2.1.0
   Compiling spin v0.5.2
   Compiling untrusted v0.7.1
   Compiling try-lock v0.2.3
   Compiling tower-service v0.3.1
   Compiling version_check v0.9.3
   Compiling ucd-trie v0.1.3
   Compiling httpdate v1.0.1
   Compiling regex-syntax v0.6.25
   Compiling either v1.6.1
   Compiling serde_json v1.0.64
   Compiling matches v0.1.8
   Compiling ansi_term v0.12.1
   Compiling base64 v0.13.0
   Compiling crc32fast v1.2.1
   Compiling xmlparser v0.13.3
   Compiling adler v1.0.2
   Compiling openssl-probe v0.1.4
   Compiling tower-layer v0.3.1
   Compiling diff v0.1.12
   Compiling bitflags v1.2.1
   Compiling hex v0.4.3
   Compiling zeroize v1.3.0
   Compiling tinyvec_macros v0.1.0
   Compiling byteorder v1.4.3
   Compiling fs_extra v1.2.0
   Compiling smallvec v1.6.1
   Compiling fastrand v1.4.1
   Compiling bit-vec v0.6.3
   Compiling md5 v0.7.0
   Compiling futures-macro v0.3.15
   Compiling tokio v1.7.1
   Compiling num-traits v0.2.14
   Compiling indexmap v1.7.0
   Compiling futures-util v0.3.15
   Compiling num-integer v0.1.44
   Compiling miniz_oxide v0.4.4
   Compiling tracing-core v0.1.18
   Compiling sharded-slab v0.1.1
   Compiling http v0.2.4
   Compiling thread_local v1.1.3
   Compiling ring v0.16.20
   Compiling pest v2.1.3
   Compiling bytes-utils v0.1.1
   Compiling form_urlencoded v1.0.1
   Compiling unicode-bidi v0.3.5
   Compiling proc-macro-error-attr v1.0.4
   Compiling proc-macro-error v1.0.4
   Compiling roxmltree v0.14.1
   Compiling pretty_assertions v0.7.2
   Compiling tinyvec v1.2.0
   Compiling jemalloc-sys v0.3.2
   Compiling regex-automata v0.1.10
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/release/build/ring-0c3d64667514526d/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("aarch64-apple-darwin")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  TARGET_CC = None
  CC = None
  CROSS_COMPILE = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/private/var/folders/s8/0tz3dy0d1sbg1_cv0ylkx_m80000gn/T/tmp27ah1k33/target/x86_64-unknown-linux-gnu/release/build/ring-1b5ef9a99e86a98a/out/aesni-x86_64-elf.o" "/Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S:1181:19: error: unexpected token in '.section' directive
  .section .note.GNU-stack,"",@progbits
                    ^
  thread 'main' panicked at 'execution failed', /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [build-s3Bam] Error 101

Expected result:

I expect sam build to correctly cross-compile a x86_64-unknown-linux-gnu Rust lambda that runs on a provided.al2 container without involving MUSL.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Apple Silicon running OSX Big Sur 11.4
2. sam --version: SAM CLI, version 1.27.2
3. AWS region: local, ap-southeast-2 otherwise

[brainstorm]

For completeness, here's the containers that I have on docker images so far:

amazon/aws-sam-cli-emulation-image-provided.al2   rapid-1.27.2                   6e864f19bc63   5 days ago     244MB
public.ecr.aws/sam/build-provided.al2             latest                         4f70a87adc2f   11 days ago    1.88GB
amazon/aws-sam-cli-emulation-image-provided.al2   latest                         70efd43277f7   11 days ago    227MB
amazon/aws-sam-cli-emulation-image-provided.al2   rapid-1.24.1                   9f3e53a81136   6 weeks ago    244MB
amazon/aws-sam-cli-emulation-image-provided.al2   <none>                         b688c0edef90   7 weeks ago    227MB
amazon/aws-sam-cli-emulation-image-python3.8      rapid-1.22.0.dev202104291816   857b0163bb0b   2 months ago   516MB
amazon/aws-sam-cli-emulation-image-python3.8      latest                         a2d941bb092d   2 months ago   500MB

But it doesn't matter because after wiping them all:

docker images | grep aws | awk -P '{ print $3 }' | xargs docker rmi

and rerunning sam build so that it only picks up the provided.al2:latest container, the results show above remain the same:

aws_lambda_builders.workflows.custom_make.make.MakeExecutionError: Make Failed: make: cargo: Command not found

Which leads me to believe that passing environment variables might wipe or mangle $PATH too within the container?

[jfuss]

@brainstorm sam build is not the same thing as sam build -u. sam build without the -u runs on your machine not a docker image. So I assume you are not seeing Cargo: Command not found when running sam build because you have Cargo installed.

In your case, the public.ecr.aws/sam/build-provided.al2 is use when using -u (this depend on what the function runtime is), which does not include Cargo. This is by design because we cannot add every language package manager/tooling into the container for build. That would make the image huge and impact performance. You can do the following to overcome this:

1. In your Makefile, install Cargo.
2. Create a new image that has Cargo installed and pass the new image to --build-image

1 will work but you will be installing Cargo over and over again for each build. Which will just make your build slower. So I recommend you to take the second path.

[brainstorm]

Thanks @jfuss, I understand the issue better now, although I see that it'll get a bit more complex before it gets easier?:

index 09a3175..7515a02 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,7 @@
 build-s3Bam:
 # TODO: Put conditional code depending on arch... for now it will lead to Exec error anyway though unless it's x86_64 provided.al2 :/
 # Intel GNU
+       curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
        cargo build --release --target x86_64-unknown-linux-gnu
        cp ./target/x86_64-unknown-linux-gnu/release/bootstrap $(ARTIFACTS_DIR)
 #

I'd like to build on top of provided.al2, but perhaps that container is a bit... hardened and not suitable for building? I'll have to check upstream's provided.al2 dockerfile, I guess, or switch to a more build-friendly container? The (dropbear? bash?) shell does not seem to play well with the cargo setup script:

% sam build -u -bi public.ecr.aws/sam/build-provided.al2
Starting Build inside a container
Building codeuri: /Users/rvalls/dev/umccr/s3-rust-noodles-bam runtime: provided.al2 metadata: {'BuildMethod': 'makefile'} functions: ['s3Bam']

Fetching public.ecr.aws/sam/build-provided.al2 Docker container image......
Mounting /Users/rvalls/dev/umccr/s3-rust-noodles-bam as /tmp/samcli/source:ro,delegated inside runtime container

Build Failed
Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : /tmp/samcli/artifacts
Error: CustomMakeBuilder:MakeBuild - Make Failed: /usr/bin/head: error reading ‘/proc/self/exe’: Bad file descriptor
/usr/bin/head: failed to close ‘/proc/self/exe’: Bad file descriptor
rustup: unknown platform bitness
/usr/bin/sh: line 358: [: : integer expression expected
info: downloading installer
rustup: Unable to run interactively. Run with -y to accept defaults, --help for additional options
make: *** [build-s3Bam] Error 1

Also, how can I define a build-only target on the same Makefile without interfering with the default sam build host-only target? I'm thinking about:

 % git diff Makefile
diff --git a/Makefile b/Makefile
index 09a3175..614bb06 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,9 @@
+build-only-container:
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+ # TODO: Other build-only related commands
+        cargo build --release --target x86_64-unknown-linux-gnu
+        cp ./target/x86_64-unknown-linux-gnu/release/bootstrap $(ARTIFACTS_DIR)
# Host-only sam build target
 build-s3Bam:
        cargo build --release --target x86_64-unknown-linux-gnu
        cp ./target/x86_64-unknown-linux-gnu/release/bootstrap $(ARTIFACTS_DIR)

Or perhaps I should drop sam build -bi and build the container separatedly outside the SAM CLI?

See https://github.com/umccr/s3-rust-noodles-bam/blob/d93d79a7e9d78e54bfb0cf141350f28986e26463/Makefile for more context.

[brainstorm]

Also, is it possible to pass a --volumes equivalent to sam build so I can source scripts and run them on the container context?

[jfuss]

I'd like to build on top of provided.al2, but perhaps that container is a bit... hardened and not suitable for building?

Not sure what you mean by this. The build-provided.al1 image should be a good base. You may need to install tools you need into the image though.

Also, how can I define a build-only target on the same Makefile without interfering with the default sam build host-only target?

Not sure I understand the question here. Are you trying to build some functions locally, while others in a container? If so that is not supported. #1825 is similar to this and I don't think we have a specific issue to cover this use-case yet.

Also, is it possible to pass a --volumes equivalent to sam build so I can source scripts and run them on the container context?

No. You can provide a customer image but we don't allow mounting of volumes into the build container.

[brainstorm]

Not sure what you mean by this. The build-provided.al1 image should be a good base. You may need to install tools you need into the image though.

Sorry, I'll try to be more specific: provided.al1 images have a too old GLIBC for Rust lambdas, so I strongly prefer provided.al2 for anything Rust... also where can I pull that build-provided images from? public.ecr.aws/lambda/*?

I have been looking around the AWS ECR public registry here:

https://gallery.ecr.aws/?operatingSystems=Linux&architectures=x86-64%2Cx86&searchTerm=lambda

Not sure I understand the question here. Are you trying to build some functions locally, while others in a container? If so that is not supported. #1825 is similar to this and I don't think we have a specific issue to cover this use-case yet.

Apologies again, what I meant was whether there was a possibility to ship the container building "recipe"/Dockerfile along with the SAM cli project itself, as in:

make build <--- builds the container using the Makefile from SAM (pointed by template.yml)
make <--- builds the binaries/artifacts for SAM (default target)

But nevermind about this question, I guess that such a building container needs to be built separatedly now.

I'll now try and build a container with the docs I referred to above in https://github.com/aws/aws-lambda-base-images and report back, thanks for the support and guidance, @jfuss.