Darwin/ARM64 tracking issue

[Keno]

I figured it would be worth having a single issue to track all the known issues on Apple Silicon. I'll try to keep this list updated as things get fixed or people encounter additional issues.

• Add MacOS(:aarch64) as a valid platform (BinaryPlatforms: Recognize MacOS(:aarch64) Pkg.jl#1916)
• Port Mach exception handling (WIP: Some minimal support for Apple Silicon #36592)
• Unconditioanlly enable CRC32 (Darwin/ARM64: Set default mcpu target #36624) Hook up ARM feature detection (via sysctl hw.optional)
• Figure out where to get a Fortran compiler from (see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96168 for gcc discussion - other compiler seem farther away)
• Build BB shard (should be relatively straightforward once we have a Fortran compiler - Add new GCC shard for iains' GCC port to Darwin Aarch64 JuliaPackaging/Yggdrasil#1626)
• Fix synchronous memory exception delivery (Darwin/ARM64: Fix throwing of synchronous memory exceptions #36625)
• Upgrade config.sub (fixed by Update our vendored copy of config.sub #36615)
• GMP fails to compile (fixed by Add GMP patch to fix build on Apple Silicon #36616)
• PCRE2 crashes (https://bugs.exim.org/show_bug.cgi?id=2618 Apple Bigsur support zherczeg/sljit#90) (Update PCRE2 to 10.36 #39310)
• uv_cpu_info errors (tracked at uv_cpu_info fails on Darwin ARM64 libuv/libuv#2911)
• LLVM9 process ARM64 relocations incorrectly (seems to be fixed in LLVM10 - needs upgrade LLVM to 10 #35318)
• Our libosxunwind is 7 years old and does not have ARM64 support - we should replace it by LLVM libunwind (issue Replace libunwind + libosxunwind with LLVM's libunwind #30154)
• Unknown LLVM issue with the following symptom (might be a general LLVM10 issue?): ([LLVM] add patches for JuliaLang/julia#36062 JuliaPackaging/Yggdrasil#1318)

      From worker 14:	While deleting: i8* %splitgep
      From worker 14:	An asserting value handle still pointed to this value!
      From worker 14:	UNREACHABLE executed at /Users/julia/julia/deps/srccache/llvm-10.0.0/lib/IR/Value.cpp:917!
      From worker 14:	
      From worker 14:	signal (6): Abort trap: 6
      From worker 14:	in expression starting at /Users/julia/julia/usr/share/julia/stdlib/v1.6/LinearAlgebra/test/diagonal.jl:11

• Some sort of (intermittent) issue during precompile:

Generating REPL precompile statements... 22/28ERROR: LoadError: IOError: stream is closed or unusable

• Test failure in worlds test:

worlds                             (4) |         failed at 2020-11-13T00:31:04.270
On worker 4:
BoundsError: attempt to access 3-element BitVector at index [0:3]

• Test failure in numbers test (related to SIGFPE handling) (Darwin/ARM64: Fix jl_throw_in_ctx #39894):

Worker 6 terminated.
numbers                            (6) |         failed at 2020-11-13T00:31:34.703
ProcessExitedException(6)

• Segfault in complex test:
  complex (2) | started at 2020-11-13T00:39:12.332

      From worker 2:	
      From worker 2:	signal (11): Segmentation fault: 11
      From worker 2:	in expression starting at /Users/julia/julia23/test/complex.jl:30
      From worker 2:	jl_method_error_bare at /Users/julia/julia23/usr/lib/libjulia.1.6.dylib (unknown line)
      From worker 2:	jl_method_error at /Users/julia/julia23/usr/lib/libjulia.1.6.dylib (unknown line)
      From worker 2:	jl_apply_generic at /Users/julia/julia23/usr/lib/libjulia.1.6.dylib (unknown line)
      From worker 2:	do_call at /Users/julia/julia23/usr/lib/libjulia.1.6.dylib (unknown line)

• Test failures in complex test (filed as Complex signed NaN test fails on Darwin/Aarch64 #38419)
• Several tests run forever:

LinearAlgebra/triangular (running for 61 minutes)
LinearAlgebra/addmul (running for 55 minutes)
bitarray (running for 53 minutes)
iterators (running for 52 minutes)
ccall (running for 39 minutes)
loading (running for 39 minutes)
sorting (running for 24 minutes)

• Test failure in inference

compiler/inference                 (5) |         failed at 2020-11-13T01:24:18.980
Test Failed at /Users/julia/julia23/test/compiler/inference.jl:944
  Expression: break_21369()
    Expected: ErrorException
      Thrown: BoundsError

• Build system hacks since we don't have a native GCC toolchain built ([Darwin/AArch64] Force BB triplet for now #38421)
• Some sort of segfault in debug build (filed as Bad LLVM code generation on Aarch64 Darwin #39818):

signal (11): Segmentation fault: 11
in expression starting at REPL[1]:1
jfptr_LinearIndices_7740 at /Users/julia/julia-master/usr/lib/julia/sys-debug.dylib (unknown line)
_jl_invoke at /Users/julia/julia-master/src/gf.c:2223
jl_apply_generic at /Users/julia/julia-master/src/gf.c:2424
ssa_substitute_op! at ./compiler/ssair/inlining.jl:1432
ssa_substitute! at ./compiler/ssair/inlining.jl:1406 [inlined]
ir_inline_item! at ./compiler/ssair/inlining.jl:369
batch_inline! at ./compiler/ssair/inlining.jl:553

• Address space related LLVM assertion (https://reviews.llvm.org/D61259#2585835)
• The linker corrupts our sysimage (Darwin/ARM64: Linker corrupts generated code #39820)

signal (11): Segmentation fault: 11
in expression starting at none:0
<= at ./int.jl:444 [inlined]
>= at ./operators.jl:409 [inlined]
unitrange_last at ./range.jl:359 [inlined]
UnitRange at ./range.jl:354 [inlined]

• Unwinding from JIT frames doesn't work (Stack traces short/missing on Darwin/ARM64 #39986)
• Integer printing is broken under make debug (DARWIN/ARM64: Bad codegen for sign extend #39823)

julia> typemin(Int32)
2147483648

• LLVM Assertion failure in iterators/bitarray test

      From worker 4:	Assertion failed: (isInt<33>(Addend) && "Invalid page reloc value."), function encodeAddend, file /Users/julia/julia-master/deps/srccache/llvm-11.0.1/lib/ExecutionEngine/RuntimeDyld/Targets/RuntimeDyldMachOAArch64.h, line 210.
      From worker 4:	
      From worker 4:	signal (6): Abort trap: 6
      From worker 4:	in expression starting at /Users/julia/julia-master/test/iterators.jl:343
      From worker 4:	__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
      From worker 4:	Allocations: 962963350 (Pool: 962516480; Big: 446870); GC: 662
Worker 4 terminated.
iterators                          (4) |         failed at 2021-02-25T16:52:13.534

• Artifacts test needs updating (Expand Artifacts tests for new platforms #39829)
• LibCURL tests fail:

LibCURL                           (24) |         failed at 2021-02-25T17:03:28.235
Error During Test at /Users/julia/julia-master/usr/share/julia/stdlib/v1.7/LibCURL/test/runtests.jl:34
  Got exception outside of a @test
  SSL peer handshake failed, the server most likely requires a client certificate to connect while requesting https://github.com/JuliaWeb/LibCURL.jl/blob/master/README.md
Test Failed at /Users/julia/julia-master/usr/share/julia/stdlib/v1.7/LibCURL/test/ssl.jl:32
  Expression: res == CURLE_OK
   Evaluated: 0x00000023 == 0x00000000

• Segfault in SVD test SVD test segfaults on Apple M1 #41440
• Darwin/ARM64: Julia freezes on nested @threads loops

[yuyichao]

Is the compiler enabling all the features available by default?

In another word, does it pass

julia/src/crc32c.c

Line 328 in a23a4ff

# ifdef __ARM_FEATURE_CRC32

by default? Or do we have to do a +crc one way or another ourselves.

[Keno]

Here's what's enabled by default:

#define __ARM64_ARCH_8__ 1
#define __ARM_64BIT_STATE 1
#define __ARM_ACLE 200
#define __ARM_ALIGN_MAX_STACK_PWR 4
#define __ARM_ARCH 8
#define __ARM_ARCH_ISA_A64 1
#define __ARM_ARCH_PROFILE 'A'
#define __ARM_FEATURE_CLZ 1
#define __ARM_FEATURE_CRYPTO 1
#define __ARM_FEATURE_DIRECTED_ROUNDING 1
#define __ARM_FEATURE_DIV 1
#define __ARM_FEATURE_FMA 1
#define __ARM_FEATURE_IDIV 1
#define __ARM_FEATURE_LDREX 0xF
#define __ARM_FEATURE_NUMERIC_MAXMIN 1
#define __ARM_FEATURE_UNALIGNED 1
#define __ARM_FP 0xE
#define __ARM_FP16_ARGS 1
#define __ARM_FP16_FORMAT_IEEE 1
#define __ARM_NEON 1
#define __ARM_NEON_FP 0xE
#define __ARM_NEON__ 1
#define __ARM_PCS_AAPCS64 1
#define __ARM_SIZEOF_MINIMAL_ENUM 4
#define __ARM_SIZEOF_WCHAR_T 4

Since I doubt there'll be a mac without crc32, we should just add that to the default feature flags in our Makefile. For everything else we can do runtime detection with sysctl.

[yuyichao]

I'm surprised that it enables crypto but not crc.... Yeah, I don't think it's worth doing a runtime detection here.

And from #36592 (comment) it doesn't seem to provide all the features that LLVM may use

The features detectable currently appears to be

hw.optional.neon_fp16: fullfp16
hw.optional.armv8_1_atomics: lse
hw.optional.armv8_crc32: crc
hw.optional.armv8_2_fhm: fp16fml
__ARM_FEATURE_CRYPTO (compile time): aes, sha2

The ones that should be supported on that CPU (all requirement from armv8.3-a) are jsconv, complxnum, rcpc, ccpp, rdm. Some of the floating point ones are quite intereting.

Also intereting that since fp16fml is reported the featureset is closer to that of a13 than a12. (that or the LLVM feature set for a12 is wrong...)

---

Anyway, this is probably a low priority item...

[Keno]

Looks like they're just shipping an old LLVM, e.g. if I try to build jsconv (just to see whether it would run) fatal error: error in backend: Cannot select: intrinsic %llvm.aarch64.fjcvtzs

[yuyichao]

Huh, which LLVM version do they have? Over at

julia/src/features_aarch64.h

Line 24 in a23a4ff

JL_FEATURE_DEF(jsconv, 13, 0) // HWCAP_JSCVT. Required in ARMv8.3

I was assuming as long as the feature is available in AArch64.td it's usable... Is that not the case? (and/or is that a mac only problem?)

[Keno]

Huh, which LLVM version do they have

I don't know. It claims to be LLVM 12, but Apple lies about versions. I'm building upstream clang now to try it out.

[yuyichao]

It also seems that although the feature was added in https://reviews.llvm.org/D54633 which is in LLVM 8.0 the intrinsic wasn't added until https://reviews.llvm.org/D64495 much later. Does that error mean that it's a recognized intrinsic but just isn't supported by the backend? I guess just writing inline assembly shoud be good enough for testing.

[Keno]

Fails upstream too.

[Keno]

Works with raw llc and +mattr though, so I'm gonna say it does exist.

[yuyichao]

... I thought the error you got is a backend one..... (so llc should behave the same as clang = = ....., unless clang emits the wrong IR...)

[Keno]

I manually added the correct mattr to llc. I also managed to get it to work with -mcpu=apple-a12 at the clang level (appears to default to apple-a7). I filed an issue with Apple to get a better error message as well as bumping the default.

[yuyichao]

Ah, OK. So you didn't set the target when running with clang.