Apple Silicon toolchain support

[jez]

I just got an M1 MacBook Pro today, and am looking into how to use this project to generate arm64 binaries (for the record: everything works fine using this project to generate x86_64 binaries, which then run under Rosetta).

In the Apple Developer Documentation, they make it out to be as simple as passing a -target flag to the C compiler, though I'm sure it'll be more work to do the same thing in Bazel.

https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary

Have anyone put thought or time into how this project might be extended to support generating arm64 binaries on M1 Macs? I'm probably going to be spending some time getting this working, and I'd love any tips, ideas, or places to start.

[rrbutani]

I thought about this a bit while working on #85.

I think there are actually two separate things:

• using clang compiled for x86_64 to produce binaries for arm64 (i.e. cross compiling)
• using clang compiled for arm64 to produce binaries for arm64

I think the first can be done with the feature #85 adds but I also think it's really the second thing that you'd want when running macOS on ARM (so that clang doesn't needlessly have to run through Rosetta).

---

To add support for arm64 darwin host platforms, we'd need to:

• update parts of configure.bzl that hardcode the CPU constraints to x86_64
  • we can either parameterize these over the current CPU or just add entries for every host CPU that we want to support (as we do for OSes – we always have macOS and linux toolchains regardless of the OS of the host platform; we let toolchain resolution pick the right one)
  • I'm inclined to just add entries for all the host CPUs we want to support since I don't know of a good way to determine the CPU type of the host from a repo rule (an alias on a select on the CPU or just running arch are the two ways that come to mind, both of which seem bad); users who aren't using toolchain resolution yet won't have their setups broken since we can add the appropriate entries to the toolchain_suite
    • this has the (odd but not bad) side-effect of letting you "choose" to use x86_64 clang on an M1 mac by running bazel under Rosetta, I think
• update cc_toolchain_config to take a host_cpu arg
  • at a glance, I think this only really affects the host_system_name, and the default target_system_name, target_cpu, abi_version, and abi_libc_version
• sysroot.bzl might need some changes depending on whether the sysroots for arm64 macOS installs are any different
• have llvm_distributions.bzl grow entries for arm64 macOS (more on this below)
• change BUILD.tpl to match configure.bzl

And I think that should be about it.

I think the bigger obstacle right now is that LLVM doesn't publish binaries for arm64 macOS (even though it's definitely possible to build LLVM for arm64 macOS). This effectively means that you'd have to provide your own download URL to to llvm_toolchain.

Adding tests for this in CI will also be tricky since (afaik) GitHub Actions doesn't yet have ARM worker machines let alone macOS ARM worker machines but that's okay.

Another question is whether we'd want to support creating universal binaries on macOS; I think it makes sense not to default to doing so (users can opt into this by setting some --copts or adding some extra_compile_flags with the features in #85 for now; eventually we can make it an option on llvm_toolchain if there's interest).

---

@jez I'm happy to try to put together these changes if you're willing to test (I don't have access to an arm64 macOS machine).

[jez]

I’m more than happy to test! Do you have a sense of when you’d have time to work on this?

Also: I think that even being able to produce arm64 binaries would be an improvement, even if they were built by x86_64 clang. I see that #85 is a draft—is that something you’d like me to test? Or is there something else holding it back from landing?

[rrbutani]

Okay! I put together a thing that uses #85 to set up a toolchain for arm64 (still x86_64 based so it's cross-compiling).

It took a little doing; I'm not on macOS 11 so my sysroot didn't have the right stuff and it took me a while to realize that the ld that ships in LLVM releases doesn't have support for the tbd (TAPI) files that newer macOS SDKs ship. I ended up having to use lld as the linker; I'm not sure this is totally right:

        # newer macOS SDKs use `.tbd` files in their sysroots; `lld` has support for this
        # (on recent versions) but `ld` (that ships in the LLVM releases) does not
        #
        # so, we need to specify that we want to use `lld`
        #
        # historically, we have not done this for macOS by default because of incomplete
        # support for Mach-O in `lld` but newer version seem to have good support.
        "extra_linker_flags": ["-fuse-ld=lld"],

I'd love to know what Apple clang does (just the output of g++ -v -xc++ - <<<"int main() { }" on an M1 machine should give it away I think).

But anyways, with the above, it does get all the way through linking on my x86_64 macOS machine. I can't actually run the generated binaries but hopefully they do actually work 🤞.

I've attached the workspace I put together to this comment.

• bazel run //:test --config=x86 to check that the regular old x86_64 -> x86_64 toolchain works, etc.
• bazel build //:test --config=arm64 to make the x86_64 -> arm64 toolchain can generate binaries
• bazel run //:test --config=arm64 to make sure the binaries emitted can actually run on arm64

In theory running bazel run //:test (without any --config to manually set the target platform) should pick up the arm64 toolchain and use it, depending on what constraints Bazel gives M1 machines. I have no idea how this interacts with Rosetta though; bazel build //:test --toolchain_resolution_debug should give us some hints.

workspace.zip

[rrbutani]

@jez re: your other questions:

I should have time to try to get the arm64 -> arm64 toolchain working over the weekend; assuming ^ works, that should be fairly straight-forward. The trickiest part will probably be finding/building an arm64 LLVM toolchain to use.

#85 is still a draft because it builds on #75 which hasn't yet been merged and because it's missing some docs and polish; I wanted to solicit some feedback about some of the design choices in that PR before cleaning it up. It is more or less functional though.

[jez]

I'd love to know what Apple clang does (just the output of g++ -v -xc++ - <<<"int main() { }" on an M1 machine should give it away I think).

Here's the output:

output.log

I'll test that workspace out now and see what happens.

[rrbutani]

Here's the output:

output.log

Thanks! Can you run /Library/Developer/CommandLineTools/usr/bin/ld -v also? I'm pretty sure it's just ld64 but just in case.

[jez]

❯ /Library/Developer/CommandLineTools/usr/bin/ld -v
@(#)PROGRAM:ld  PROJECT:ld64-650.9
BUILD 13:09:13 May 28 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 12.0.5, (clang-1205.0.22.11) (static support for 27, runtime is 27)
TAPI support using: Apple TAPI version 12.0.5 (tapi-1205.0.7.1)

[jez]

is the @macos-11.3-sdk// repo basically the same set of files I have at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk, judging from the output.log above?

[rrbutani]

yup, exactly

[rrbutani]

I had to grab it externally because I'm on an older version of macOS that doesn't have an SDK with the right stuff to build for arm64

but for host arm64 toolchains (as in arm64 -> arm64) we shouldn't actually need to grab it; like we do with the other host toolchains, we can just assume what's on the host system actually works when you're targeting the host system

[jez]

~/stripe/sandbox/rrbutani-workspace   19s
❯ ./bazel run //:test --config=x86
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel/e9cb4153e0861999826e8879b02ae2cc/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin'
ERROR: Analysis of target '//:test' failed; build aborted: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Elapsed time: 69.390s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (16 packages loaded, 61 targets configured)
FAILED: Build did NOT complete successfully (16 packages loaded, 61 targets configured)

~/stripe/sandbox/rrbutani-workspace
❯ ./bazel --version
bazel 4.2.1

(I had to make a slight change to the repo, which is to make it use a script that has contents identical to this:

https://github.com/jez/ragel-bison-parser-sandbox/blob/master/bazel

because our company laptops prevent us from installing bazelisk) but otherwise the above is the result of running things.

Interestingly enough, that's the same error I get when trying to build a normal bazel project on my macbook. For example this tiny project shows the same problems.

bazel-test.zip

[rrbutani]

hmm

Does /private/var/tmp/_bazel/e9cb4153e0861999826e8879b02ae2cc/external/local_config_cc/BUILD:48 look like:

cc_toolchain_suite(
    name = "toolchain",
    toolchains = {
        "k8|clang": ":cc-clang-linux",
        "darwin|clang": ":cc-clang-darwin",
        "k8": ":cc-clang-linux",
        "darwin": ":cc-clang-darwin",
    },
)

for you?

[rrbutani]

oh whoops, nvm; I totally missed that that's @local_config_cc

[rrbutani]

It's not clear to me why it's even analyzing @local_config_cc//:toolchain; can you post what bazel build //:test --toolchain_resolution_debug prints out?

[jez]

It's not clear to me why it's even analyzing @local_config_cc//:toolchain; can you post what bazel build //:test --toolchain_resolution_debug prints out?

~/stripe/sandbox/rrbutani-workspace
❯ bazel build //:test --toolchain_resolution_debug
INFO: Build options --platforms and --toolchain_resolution_debug have changed, discarding analysis cache.
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @llvm_toolchain//:cc-clang-linux; mismatching values: linux
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: execution @local_config_platform//:host: Selected toolchain @llvm_toolchain//:cc-clang-darwin
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain //:clang-darwin-arm64-toolchain; mismatching values: arm64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm, android
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @llvm_toolchain//:cc-clang-darwin
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host,
ERROR: /private/var/tmp/_bazel/e9cb4153e0861999826e8879b02ae2cc/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin'
ERROR: Analysis of target '//:test' failed; build aborted: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Elapsed time: 0.924s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 60 targets configured)