7.0.0 rc1: 16 libc++ tests failing on macOS/x86_64 (Exit Code: -11)

[vedantk]

Bugzilla Link	38473
Resolution	FIXED
Resolved on	Sep 10, 2020 10:26
Version	7.0
OS	All
CC	@zmodem,@ldionne,@mclow

Extended Description

These tests all fail in what looks to be the same way:

libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/ctor_copy.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/ctor_default.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/rethrow_if_nested.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp
libc++ :: std/language.support/support.exception/propagation/make_exception_ptr.pass.cpp
libc++ :: std/language.support/support.exception/propagation/rethrow_exception.pass.cpp
libc++ :: std/thread/futures/futures.async/async.pass.cpp
libc++ :: std/thread/futures/futures.promise/dtor.pass.cpp
libc++ :: std/thread/futures/futures.promise/set_exception.pass.cpp
libc++ :: std/thread/futures/futures.promise/set_exception_at_thread_exit.pass.cpp
libc++ :: std/thread/futures/futures.shared_future/get.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/dtor.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/make_ready_at_thread_exit.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/operator.pass.cpp
libc++ :: std/thread/futures/futures.unique_future/get.pass.cpp

To reproduce the failure, I did:

$ export MACOSX_DEPLOYMENT_TARGET="10.9"
$ ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files -j20

Here's a sample failure:

FAIL: libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp (49834 of 54980)
******************** TEST 'libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp' FAILED ********************
Compiled With: ['/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-x', 'c++', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/assign.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++2a', '-include', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/include', '-I/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build', '-isysroot', '/Volumes/Xcode10A230a_m18A350_i16A343_t16J338_w16R338a_b16P349_XcodeInternals_32bitSupport_FastSim_Boost_ASan_36GB/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wno-aligned-allocation-unavailable', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-Wl,-rpath,/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem']
Command: ['/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe']
Exit Code: -11

In this case, assign.pass.cpp.exe does not exist. The directory it's supposed to be in exists and is empty.

If I strip out the single-quotes, perform the compile+link steps manually, and run assign.pass.cpp.exe it passes.

[vedantk]

assigned to @ldionne

[mclow]

I'll be looking at this this morning.

[mclow]

Odd that even though you exported MACOSX_DEPLOYMENT_TARGET="10.9", the compile command contains '-mmacosx-version-min=10.13'.

(Thanks to Louis for noticing this)

[mclow]

There's clearly multiple things going on here.

0. Error -11 is a segmentation fault.
1. The reason you can't find the programs is that lit removes them after they've been run. They were built successfully.
2. When you build them manually, and then run them, you are running against the system dylib, so you don't see the failure.
3. The programs are definitely crashing. (see below).
4. The programs are being built with a macosx target of 10.13, not 10.9

Here's how you can build (and run) just one test.

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-x' 'c++' 'rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp' '-c' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Werror=thread-safety' '-std=c++17' '-include' 'rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp' '-nostdinc++' '-Irc1/llvm.src/projects/libcxx/include' '-Irc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build' '-isysroot' '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk' '-Irc1/llvm.src/projects/libcxx/test/support' '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/usr/bin/python rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"' '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER' '-Wall' '-Wextra' '-Werror' '-Wuser-defined-warnings' '-Wshadow' '-Wno-unused-command-line-argument' '-Wno-attributes' '-Wno-pessimizing-move' '-Wno-c++11-extensions' '-Wno-user-defined-literals' '-Wno-noexcept-type' '-Wno-aligned-allocation-unavailable' '-Wsign-compare' '-Wunused-variable' '-Wunused-parameter' '-Wunreachable-code' '-Wno-conversion' '-Wno-unused-local-typedef' '-c'

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Lrc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-Wl,-rpath,rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-nodefaultlibs' '-lc++experimental' '-lc++fs' '-lc++' '-lSystem'

$ DYLD_LIBRARY_PATH=rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/lib rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe

Segmentation fault: 11

I suspect that some of the other bits - (like libcxxabi) are not built with the same options as libcxx and/or the test programs and that is causing the crashes.

[vedantk]

Thanks Marshall, I can reproduce the crash when specifying the correct DYLD_LIBRARY_PATH:

• thread #​1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  frame #​0: 0x00000001001b00cd libunwind.dylib_Unwind_RaiseException + 93 libunwind.dylib_Unwind_RaiseException:
  -> 0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14)
  0x1001b00d2 <+98>: leaq -0x398(%rbp), %rbx
  0x1001b00d9 <+105>: movq %rbx, %rdi
  0x1001b00dc <+108>: movq %r15, %rsi
  Target 0: (assign.pass.cpp.exe) stopped.
  (lldb) bt
• thread #​1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  • frame #​0: 0x00000001001b00cd libunwind.dylib_Unwind_RaiseException + 93 frame #​1: 0x0000000100190e9f libc++abi.dylib__cxa_rethrow_primary_exception + 239
    frame #​2: 0x00000001000afebc libc++.1.dylibstd::rethrow_exception(std::exception_ptr) + 12 frame #​3: 0x0000000100000ab8 assign.pass.cpp.exemain at assign.pass.cpp:53
    frame #​4: 0x00007fff57147015 libdyld.dylib`start + 1
    (lldb) image list
    [ 0] 8D2E4DC3-765E-3C89-A451-23AA36AFB082 0x0000000100000000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe
    [ 1] 8A72DE9C-A136-3506-AA02-4BA2B82DCAF3 0x0000000100003000 /usr/lib/dyld
    [ 2] 053A1AC0-ED15-38D8-8B05-174BDAEC9B72 0x00000001000a1000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++.1.dylib
    [ 3] CD555F3B-FDDB-35E5-A2FB-FBBF3D62031A 0x00007fff54e60000 /usr/lib/libSystem.B.dylib
    [ 4] 20AF2CAA-9E97-3FB5-9BCE-A4E793429348 0x0000000100173000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++abi.dylib
    ...
    [ 37] 1E105381-88F9-3296-813F-A1E9B14BD93C 0x00000001001a8000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libunwind.dylib

[vedantk]

Could the GPFLT be due to an alignment exception?

The crashing instruction appears to be:

0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14)

I think that requires 16 byte alignment, but 0x10(%r14) isn't sufficiently aligned?

(lldb) reg read xmm0
xmm0 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
(lldb) p/x $r14+0x10
(unsigned long) $2 = 0x00000001003002e8

[ldionne]

I'm starting to suspect this has something to do with the changes we did to disable the use of aligned allocation functions when the deployment target does not support them. Maybe there's something wrong with that change. I'll try to reproduce using ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files and report when I'm able to repro.

[ldionne]

Minimal reproduction script
This has nothing to do with the bootstrapping process. I can reproduce with a system compiler and the RC1 sources.

[ldionne]

I can now also reproduce on ToT, but one has to include libunwind in the set of built projects, which I was not doing initially (using the monorepo setup).

[ldionne]

Updated reproduction script
Okay, so with Marshall's help, I was able to narrow it down to a single revision on libunwind:

r291830 (around Release 5) => SEGFAULT
r276215 => SEGFAULT
r276214 => no SEGFAULT

In other words, r276215 introduces the failure. I also attached the script I used to reproduce. My setup was a checkout of LLVM trunk with projects/libcxx on trunk, projects/libcxxabi on trunk, and projects/libunwind on the various revisions shown above.

[ldionne]

This has been failing for a long time, but I don't know why we only caught it recently. It is not impossible that we were picking up the system's libunwind, which may be older than that revision of libunwind.

[ldionne]

It's possible to reproduce on Trunk with our current lit test suite. It suffices to make sure that libunwind is being built ninja -C build unwind to trigger the issue.

Also note that configuring CMake with -DLIBCXXABI_USE_LLVM_UNWINDER=ON seems to solve the problem (that option defaults to OFF).

[ldionne]

With -DLIBCXXABI_USE_LLVM_UNWINDER=OFF, the command line used to build cxa_exception.cpp is:

$CXX  -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp

and unwind.h is: $SDKROOT/usr/include/unwind.h

With -DLIBCXXABI_USE_LLVM_UNWINDER=ON, the command line is:

$CXX -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libunwind/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC   -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp

and unwind.h is /Users/ldionne/work/llvm/libunwind/include/unwind.h. Notice how -DLIBCXXABI_USE_LLVM_UNWINDER=ON causes -I/Users/ldionne/work/llvm/libunwind/include to be added to the header search paths.

In other words, without -DLIBCXXABI_USE_LLVM_UNWINDER=ON, we're using the system's unwind.h, to build libc++abi, and so we're using struct _Unwind_Exception provided by the system's unwind.h, which has 8 bit alignment (because it predates r276215, where the alignment was changed). However, when we link against LLVM's libunwind.dylib (using DYLD_LIBRARY_PATH as in the reproduction), we're linking against a libunwind.dylib that thinks the alignment of struct _Unwind_Exception is 16 bits, because it was built with the new headers.

This is where we get a mismatch: libc++abi thinks the alignment of struct _Unwind_Exception is 8, but libunwind.dylib think it is 16.

As soon as we go back to using the system's libunwind.dylib (which also thinks alignment is 8), the problem goes away. Similarly, when we start building libc++abi.dylib with the headers that know the alignment to be 16, the problem goes away.

[ldionne]

I think we understand the problem pretty well at this point. Marshall and I have discussed the following solutions:

1. When building libunwind, use the target system's unwind.h headers. This means that we'll build and ship a libunwind.dylib that's compatible with the headers present on the system. This also means that libc++abi will have to be built against those system headers, otherwise we'll run into the issue encountered here, but reversed (libc++abi will think 16 bytes alignment, but libunwind will think 8 bytes, which is technically OK but let's not go there). The downside of this approach is that we're not shipping a release that's self consistent: the unwind.h header shipped with LLVM 7 should never be used, because the libunwind.dylib will have been built with different and incompatible headers.

2. Build libc++abi AND libunwind against the LLVM 7 unwind.h headers. This means that we'd ship libc++.dylib/libc++abi.dylib/libunwind.dylib that are consistent with the headers provided with LLVM 7, but not with the system's headers. The downside is that a program using the system's unwind.h headers could NOT be linked against libc++abi.dylib/libunwind.dylib shipped with LLVM 7, because those would have been built with newer and incompatible libunwind headers.

We currently lean towards option (1) for lack of a better solution.

[zmodem]

We currently lean towards option (1) for lack of a better solution.

Is there a patch or something for this?

[ldionne]

Not yet. I've been distracted by other work and you mentioned this was not a release blocker for the moment, so I set this aside. Please LMK if you feel like this needs to be resolved for the release and I'll try to re-prioritize.

[zmodem]

Not yet. I've been distracted by other work and you mentioned this was not a
release blocker for the moment, so I set this aside. Please LMK if you feel
like this needs to be resolved for the release and I'll try to re-prioritize.

Right, let's not wait for this.

[ldionne]

Since then, Apple has updated the version of libunwind (both the headers and the dylib) shipped in SDKs. I don't think this issue is still relevant. Vedant, please re-open if you disagree.

[llvmbot]

mentioned in issue llvm/llvm-bugzilla-archive#39051

[llvmbot]

mentioned in issue llvm/llvm-bugzilla-archive#39051