Build broken on Apple Silicon computers (and/or Graviton2 builds)

[brainstorm]

Hello @nmoutschen, branching out our discussions over here from https://twitter.com/braincode/status/1443549985803431947...

Also, btw, this is an issue I originally reported upstream on briansmith/ring#1332

$ git clone https://github.com/aws-samples/serverless-rust-demo.git
Cloning into 'serverless-rust-demo'...
remote: Enumerating objects: 43, done.
remote: Counting objects: 100% (43/43), done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 43 (delta 13), reused 34 (delta 10), pack-reused 0
Receiving objects: 100% (43/43), 14.43 KiB | 3.61 MiB/s, done.
Resolving deltas: 100% (13/13), done.
$ cd serverless-rust-demo
$ make build
cargo build --release --target aarch64-unknown-linux-gnu
    Updating crates.io index
    Updating git repository `https://github.com/awslabs/aws-sdk-rust`
   Compiling libc v0.2.103
   Compiling proc-macro2 v1.0.29
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.77
   Compiling autocfg v1.0.1
   Compiling cfg-if v1.0.0
   Compiling log v0.4.14
   Compiling pin-project-lite v0.2.7
   Compiling once_cell v1.8.0
   Compiling lazy_static v1.4.0
   Compiling memchr v2.4.1
   Compiling bytes v1.1.0
   Compiling smallvec v1.7.0
   Compiling futures-core v0.3.17
   Compiling itoa v0.4.8
   Compiling parking_lot_core v0.8.5
   Compiling scopeguard v1.1.0
   Compiling proc-macro-hack v0.5.19
   Compiling futures-channel v0.3.17
   Compiling proc-macro-nested v0.1.7
   Compiling futures-task v0.3.17
   Compiling futures-sink v0.3.17
   Compiling pin-utils v0.1.0
   Compiling slab v0.4.4
   Compiling futures-io v0.3.17
   Compiling ryu v1.0.5
   Compiling fnv v1.0.7
   Compiling hashbrown v0.11.2
   Compiling httparse v1.5.1
   Compiling tower-service v0.3.1
   Compiling try-lock v0.2.3
   Compiling percent-encoding v2.1.0
   Compiling httpdate v1.0.1
   Compiling cc v1.0.70
   Compiling spin v0.5.2
   Compiling untrusted v0.7.1
   Compiling semver v1.0.4
   Compiling either v1.6.1
   Compiling regex-syntax v0.6.25
   Compiling ppv-lite86 v0.2.10
   Compiling base64 v0.13.0
   Compiling anyhow v1.0.44
   Compiling serde_derive v1.0.130
   Compiling serde v1.0.130
   Compiling tower-layer v0.3.1
   Compiling zeroize v1.4.2
   Compiling matches v0.1.9
   Compiling serde_json v1.0.68
   Compiling openssl-probe v0.1.4
   Compiling hex v0.4.3
   Compiling async-trait v0.1.51
   Compiling fixedbitset v0.2.0
   Compiling remove_dir_all v0.5.3
   Compiling crossbeam-utils v0.8.5
   Compiling unicode-segmentation v1.8.0
   Compiling ansi_term v0.12.1
   Compiling multimap v0.8.3
   Compiling fastrand v1.5.0
   Compiling urlencoding v1.3.3
   Compiling xmlparser v0.13.3
   Compiling instant v0.1.11
   Compiling tracing-core v0.1.20
   Compiling sharded-slab v0.1.3
   Compiling thread_local v1.1.3
   Compiling indexmap v1.7.0
   Compiling tokio v1.12.0
   Compiling futures-macro v0.3.17
   Compiling futures-util v0.3.17
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling lock_api v0.4.5
   Compiling http v0.2.5
   Compiling itertools v0.10.1
   Compiling bytes-utils v0.1.1
   Compiling form_urlencoded v1.0.1
   Compiling ring v0.16.20
   Compiling heck v0.3.3
   Compiling regex v1.5.4
   Compiling regex-automata v0.1.10
   Compiling want v0.3.0
   Compiling tracing-log v0.1.2
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/rvalls/dev/umccr/serverless-rust-demo/target/release/build/ring-5100f9369a354546/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("3")
  TARGET = Some("aarch64-unknown-linux-gnu")
  HOST = Some("aarch64-apple-darwin")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = Some("aarch64-linux-gnu-gcc")
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = None

  --- stderr
  running "aarch64-linux-gnu-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/Users/rvalls/dev/umccr/serverless-rust-demo/target/aarch64-unknown-linux-gnu/release/build/ring-5243877d4a8e9964/out/aesv8-armx-linux64.o" "/Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesv8-armx-linux64.S"
  thread 'main' panicked at 'failed to execute ["aarch64-linux-gnu-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/Users/rvalls/dev/umccr/serverless-rust-demo/target/aarch64-unknown-linux-gnu/release/build/ring-5243877d4a8e9964/out/aesv8-armx-linux64.o" "/Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesv8-armx-linux64.S"]: No such file or directory (os error 2)', /Users/rvalls/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [build] Error 101

[nmoutschen]

Hey @brainstorm !

After some experimentation on M1 macs (thanks @msailes for the help):

• cross build doesn't seem to run into a container on M1 macs
• using CC_aarch64_unknown_linux_gnu=clang cross build --target aarch64-unknown-linux-gnu fixes the No such file or directory (os error 2) issue.

However, there is another issue popping up ('assert.h' file not found) when building ring.

[nmoutschen]

For Graviton2 instances, I just tried with cross and it works fine.

[brainstorm]

From what I just learned in briansmith/ring#1332 (comment) and spack/spack#26317 (comment), this might require GCC12 to be released in order to be fixed properly for M1 machines :-S

[michaelbrewer]

For sam i have had to make a custom Makefile

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Hello World Rust Lambda function 

Resources:
  HelloRustFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: HelloRust
      Handler: bootstrap.is.real.handler
      Runtime: provided.al2
      MemorySize: 256
      CodeUri: .
      Architectures:
        - arm64
    Metadata:
      BuildMethod: makefile

Which currently just references a simple shell

build-HelloRustFunction:
	sh ./build.sh
	cp ./target/aarch64-unknown-linux-gnu/release/bootstrap $(ARTIFACTS_DIR)

The actual docker build

LAMBDA_ARCH="linux/arm64"
RUST_TARGET="aarch64-unknown-linux-gnu"
RUST_VERSION="latest"

docker run \
 --platform ${LAMBDA_ARCH} \
 --rm --user "$(id -u)":"$(id -g)" \
 -v "${PWD}":/usr/src/myapp -w /usr/src/myapp rust:${RUST_VERSION} \
 cargo build --release --target ${RUST_TARGET}

[brainstorm]

Thanks @michaelbrewer, that's pretty much what I use in:

https://github.com/umccr/s3-rust-noodles-bam

Not ideal though:

https://github.com/umccr/s3-rust-noodles-bam/blob/master/FLUKES.md

Local builds should take less that 6 minutes and, ideally, they should not involve docker at all (just regular cargo build, which takes a few seconds in a M1)... it should just work and compile aws-rust-sdk dependencies like ring, IMHO.

[nmoutschen]

I finally managed to get my hands on an M1 mac to test around. The latest release of cross (0.2.1, released in June 2020) doesn't actually run cargo in a container when run on an M1 mac.

It does work when installing it from the Git repo directly (cargo install --git https://github.com/rust-embedded/cross.git). However, it's very, very slow, so. I haven't completed a build yet.

[brainstorm]

Did you manage to reproduce my steps on briansmith/ring#1332 (comment) ? I'm curious to see if you are able to crack this ring dependency nut, IMO it's all what's needed to solve in order to have good M1-based builds sans docker/cross.

[nmoutschen]

I've got the same error message (No such file or directory (os error 2)) even on non-M1 Macs. I see that you're using aarch64-linux-gnu-gcc as compiler, but depending on what toolchain you've installed, it might have a slightly different name (e.g. aarch64-unknown-linux-gnu-gcc on my x86 mac).

That's actually one of the reasons why I prefer to use cross here: while it doesn't work well for M1 macs yet, it abstracts those small differences away so that most people can build this project no matter their setups.