[stdlib] AtomicWaitQueue crashes on Arm 32 bit

[colemancda]

Describe the bug
Swift's runtime crashes in AtomicWaitQueue.createNewQueue when accessing metadata during initialization.

To Reproduce
Steps to reproduce the behavior:

1. Run ./build.sh with SWIFT_BUILD_CONFIGURATION=Debug to cross compile swift-hello and the Swift runtime libraries for Debian 11 with debug info.
2. Run swift-hello with LLDB or GDB.

Expected behavior
Should not crash. Using the same scripts to cross compile Swift but setting SWIFT_VERSION=5.5.3 and using a Swift 5.5.3 compiler, the program does not crash. This is a regression for Swift 5.6.1.

Environment (please complete the following information):

• OS: Debian 11 Armhf (armv7)
• Swift 5.6.1 runtime cross compiled with
  https://github.com/colemancda/swift-armv7
• Swift 5.6.1 compiler from official Swift 5.6.1 Ubuntu Docker image
• Docker environment to build
  https://hub.docker.com/layers/swift-armv7/colemancda/swift-armv7/amd64-prebuilt/images/sha256-5416cdf242a46a3fa580e8b919699461137a2adf85bdf7d6f693d117adb8b6dc?context=explore

Additional context
Here is a debug stack trace with LLDB.

Process 500215 launched: '/home/coleman/Developer/swift-armv7/build/swift-hello/armv7-unknown-linux-gnueabihf/debug/swift-hello' (arm)
Process 500215 stopped
* thread #1, name = 'swift-hello', stop reason = signal SIGBUS: illegal alignment
    frame #0: 0xf7d90688 libswiftCore.so`swift::MetadataWaitQueue* swift::AtomicWaitQueue<swift::MetadataWaitQueue, swift::IndirectMutex>::Worker::createNewQueue<>() at AtomicWaitQueue.h:427:24
   424    private:
   425      template <class... Args>
   426      static Impl *createNewQueue(Args &&...args) {
-> 427        auto queue = new Impl(std::forward<Args>(args)...);
   428        queue->WaitQueueLock.lock();
   429        return queue;
   430      }
(lldb) thread backtrace
* thread #1, name = 'swift-hello', stop reason = signal SIGBUS: illegal alignment
  * frame #0: 0xf7d90688 libswiftCore.so`swift::MetadataWaitQueue* swift::AtomicWaitQueue<swift::MetadataWaitQueue, swift::IndirectMutex>::Worker::createNewQueue<>() at AtomicWaitQueue.h:427:24
    frame #1: 0xf7d905b4 libswiftCore.so`swift::MetadataWaitQueue* swift::AtomicWaitQueue<swift::MetadataWaitQueue, swift::IndirectMutex>::Worker::createQueue<>(this=0xfffeaf0c) at AtomicWaitQueue.h:260:24
    frame #2: 0xf7d90536 libswiftCore.so`swift::PrivateMetadataTrackingInfo::initial(worker=0xfffeaf0c, initialState=Allocating) at MetadataCache.h:707:63
    frame #3: 0xf7d7100c libswiftCore.so`swift::MetadataCacheEntryBase<(anonymous namespace)::GenericCacheEntry, void const*>::MetadataCacheEntryBase(this=0xf7f1f1c0, worker=0xfffeaf0c, initialState=Allocating) at MetadataCache.h:904:22
    frame #4: 0xf7d70f6c libswiftCore.so`swift::VariadicMetadataCacheEntryBase<(anonymous namespace)::GenericCacheEntry>::VariadicMetadataCacheEntryBase(this=0xf7f1f1c0, key=0xfffeac4c, worker=0xfffeaf0c, initialState=Allocating, value=0x00000000) at MetadataCache.h:1366:9
    frame #5: 0xf7d738ae libswiftCore.so`(anonymous namespace)::GenericCacheEntry::GenericCacheEntry(this=0xf7f1f1c0, key=MetadataCacheKey @ 0xfffeac4c, worker=0xfffeaf0c, request=MetadataRequest @ 0xfffeac48, description=0xf7e39b9c, arguments=0xfffeafc4) at Metadata.cpp:293:9
    frame #6: 0xf7d73822 libswiftCore.so`std::pair<(anonymous namespace)::GenericCacheEntry*, bool> swift::StableAddressConcurrentReadableHashMap<(anonymous namespace)::GenericCacheEntry, swift::TaggedMetadataAllocator<(this=0xfffeadc4, wrapper=0x004631ac, created=true)14>, swift::IndirectMutex>::getOrInsert<swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&>(swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&)::'lambda'(swift::HashMapElementWrapper<(anonymous namespace)::GenericCacheEntry>*, bool)::operator()(swift::HashMapElementWrapper<(anonymous namespace)::GenericCacheEntry>*, bool) const at Concurrent.h:1211:26
    frame #7: 0xf7d73660 libswiftCore.so`void swift::ConcurrentReadableHashMap<swift::HashMapElementWrapper<(anonymous namespace)::GenericCacheEntry>, swift::IndirectMutex>::getOrInsert<swift::MetadataCacheKey, std::pair<(anonymous namespace)::GenericCacheEntry*, bool> swift::StableAddressConcurrentReadableHashMap<(this=0xf7f1eb38, key=MetadataCacheKey @ 0xfffead58, call=0xfffeadc4)::GenericCacheEntry, swift::TaggedMetadataAllocator<(unsigned short)14>, swift::IndirectMutex>::getOrInsert<swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&>(swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&)::'lambda'(swift::HashMapElementWrapper<(anonymous namespace)::GenericCacheEntry>*, bool)>(swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker& const&) at Concurrent.h:1116:17
    frame #8: 0xf7d7344e libswiftCore.so`std::pair<(anonymous namespace)::GenericCacheEntry*, bool> swift::StableAddressConcurrentReadableHashMap<(anonymous namespace)::GenericCacheEntry, swift::TaggedMetadataAllocator<(this=0xf7f1eb38, key=MetadataCacheKey @ 0xfffeae1c, args=0xfffeaf0c, args=0xfffeaf84, args=0xfffeaf7c, args=0xfffeaf80)14>, swift::IndirectMutex>::getOrInsert<swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&>(swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&) at Concurrent.h:1204:9
    frame #9: 0xf7d730ee libswiftCore.so`std::pair<(anonymous namespace)::GenericCacheEntry*, bool> swift::LockingConcurrentMapStorage<(anonymous namespace)::GenericCacheEntry, (this=0xf7f1eb38, key=MetadataCacheKey @ 0xfffeae5c, args=0xfffeaf0c, args=0xfffeaf84, args=0xfffeaf7c, args=0xfffeaf80)14>::getOrInsert<swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&>(swift::MetadataCacheKey, swift::MetadataWaitQueue::Worker&, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&) at MetadataCache.h:184:16
    frame #10: 0xf7d72f8e libswiftCore.so`std::pair<(anonymous namespace)::GenericCacheEntry*, swift::MetadataResponse> swift::LockingConcurrentMap<(anonymous namespace)::GenericCacheEntry, swift::LockingConcurrentMapStorage<(this=0xf7f1eb38, key=MetadataCacheKey @ 0xfffeaf1c, args=0xfffeaf84, args=0xfffeaf7c, args=0xfffeaf80)::GenericCacheEntry, (unsigned short)14> >::getOrInsert<swift::MetadataCacheKey, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&>(swift::MetadataCacheKey, swift::MetadataRequest&, swift::TargetTypeContextDescriptor<swift::InProcess> const*&, void const* const*&) at MetadataCache.h:247:27
    frame #11: 0xf7d6aeb6 libswiftCore.so`_swift_getGenericMetadata(request=MetadataRequest @ 0xfffeaf84, arguments=0xfffeafc4, description=0xf7e39b9c) at Metadata.cpp:815:23
    frame #12: 0xf7d6ae22 libswiftCore.so`::swift_getGenericMetadata(request=MetadataRequest @ 0xfffeafa4, arguments=0xfffeafc4, description=0xf7e39b9c) at Metadata.cpp:827:10
    frame #13: 0xf7d3a3cc libswiftCore.so`__swift_instantiateGenericMetadata at <compiler-generated>:0
    frame #14: 0xf7bc82b8 libswiftCore.so`$sSRMa at <compiler-generated>:0
    frame #15: 0xf7db336e libswiftCore.so`swift::MetadataAccessFunction::operator(this=0xfffeb0a8, request=MetadataRequest @ 0xfffeaffc, arg0=0xf7e0d5b8)(swift::MetadataRequest, void const*) const at Metadata.h:3721:12
    frame #16: 0xf7db20ac libswiftCore.so`swift::MetadataAccessFunction::operator(this=0xfffeb0a8, request=MetadataRequest @ 0xfffeb054, args=ArrayRef<const void *> @ 0xfffeb04c)(swift::MetadataRequest, __swift::__runtime::llvm::ArrayRef<void const*>) const at Metadata.h:3700:14
    frame #17: 0xf7da6fb4 libswiftCore.so`(anonymous namespace)::DecodedMetadataBuilder::createBoundGenericType(this=0xfffedcdc, anyTypeDecl=0xf7e39b9c, genericArgs=ArrayRef<const swift::TargetMetadata<swift::InProcess> *> @ 0xfffeb134, parent=0x00000000) const at MetadataLookup.cpp:1451:12
    frame #18: 0xf7da1f54 libswiftCore.so`swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(this=0xfffedc3c, Node=0xfffee0e0, depth=1, forRequirement=false) at TypeDecoder.h:578:22
    frame #19: 0xf7da1a32 libswiftCore.so`swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(this=0xfffedc3c, Node=0xfffee0f8, depth=0, forRequirement=false) at TypeDecoder.h:508:14
    frame #20: 0xf7da146c libswiftCore.so`swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(this=0xfffedc3c, Node=0xfffee0f8, forRequirement=false) at TypeDecoder.h:477:12
    frame #21: 0xf7da13fa libswiftCore.so`swift::TypeLookupErrorOr<(anonymous namespace)::DecodedMetadataBuilder::BuiltType> swift::Demangle::__runtime::decodeMangledType<(anonymous namespace)::DecodedMetadataBuilder>(Builder=0xfffedcdc, Node=0xfffee0f8, forRequirement=false)::DecodedMetadataBuilder&, swift::Demangle::__runtime::Node*, bool) at TypeDecoder.h:1567:8
    frame #22: 0xf7d9ec14 libswiftCore.so`swift_getTypeByMangledNodeImpl(request=MetadataRequest @ 0xfffedd30, demangler=0xfffede48, node=0xfffee0f8, origArgumentVector=0x00000000, substGenericParam=swift::SubstGenericParameterFn @ 0xfffedd44, substWitnessTable=swift::SubstDependentWitnessTableFn @ 0xfffedd48)>, std::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) at MetadataLookup.cpp:1737:15
    frame #23: 0xf7d9eb54 libswiftCore.so`::swift_getTypeByMangledNode(request=MetadataRequest @ 0xfffedd98, demangler=0xfffede48, node=0xfffee0f8, arguments=0x00000000, substGenericParam=swift::SubstGenericParameterFn @ 0xfffeddb4, substWitnessTable=swift::SubstDependentWitnessTableFn @ 0xfffeddb8) at CompatibilityOverrideRuntime.def:149:1
    frame #24: 0xf7d9edda libswiftCore.so`swift_getTypeByMangledNameImpl(request=MetadataRequest @ 0xfffee778, typeName=(Data = "SRy\U00000001\xe8\xe1`", Length = 9), origArgumentVector=0x00000000, substGenericParam=swift::SubstGenericParameterFn @ 0xfffee794, substWitnessTable=swift::SubstDependentWitnessTableFn @ 0xfffee798)>, std::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) at MetadataLookup.cpp:1801:10
    frame #25: 0xf7d9d620 libswiftCore.so`::swift_getTypeByMangledName(request=MetadataRequest @ 0xfffee7f0, typeName=(Data = "SRy\U00000001\xe8\xe1`", Length = 9), arguments=0x00000000, substGenericParam=swift::SubstGenericParameterFn @ 0xfffee80c, substWitnessTable=swift::SubstDependentWitnessTableFn @ 0xfffee810) at CompatibilityOverrideRuntime.def:157:1
    frame #26: 0xf7d9d88a libswiftCore.so`::swift_getTypeByMangledNameInContext(typeNameStart="SRy\U00000001\xe8\xe1`", typeNameLength=9, context=0x00000000, genericArgs=0x00000000) at MetadataLookup.cpp:1854:10
    frame #27: 0xf7a25e34 libswiftCore.so`__swift_instantiateConcreteTypeFromMangledName at <compiler-generated>:0
    frame #28: 0xf7a46c9c libswiftCore.so`$ss12_SmallStringVyABSgSRys5UInt8VGcfC at SmallString.swift:273:14
    frame #29: 0xf786a064 libswiftCore.so`$sSS21_builtinStringLiteral17utf8CodeUnitCount7isASCIISSBp_BwBi1_tcfC at String.swift:652:19
    frame #30: 0xf70c5b84 libFoundation.so`$s10Foundation19__CFInitializeSwiftyyF + 36
    frame #31: 0xf70c5b5c libFoundation.so`__CFInitializeSwift + 12
    frame #32: 0xf72a1886 libFoundation.so`__CFInitialize at CFRuntime.c:1247:9
    frame #33: 0xf7fd0f40 ld-2.31.so`call_init(l=<unavailable>, argc=1, argv=0xfffef664, env=0xfffef66c) at dl-init.c:72:3
    frame #34: 0xf7fd0fe2 ld-2.31.so`_dl_init at dl-init.c:30:6
    frame #35: 0xf7fd0fd4 ld-2.31.so`_dl_init(main_map=0xf7fef9b8, argc=1, argv=0xfffef664, env=0xfffef66c) at dl-init.c:119
    frame #36: 0xf7fc5ac4 ld-2.31.so`_dl_start_user + 52
(lldb) 

[tbkka]

CC: @lorentey @grynspan

[grynspan]

This issue should be fixed with my changes to make this function use swift_cxx_newObject() instead of operator new. Did that change get rolled back?

[grynspan]

My change appears to still be present on main. This line from the crash:

auto queue = new Impl(std::forward<Args>(args)...);

Is now:

auto queue = swift_cxx_newObject<Impl>(std::forward<Args>(args)...);

And swift_cxx_newObject<T>() correctly allocates instances of type T when T's alignment exceeds the default.

Could it be that lowering SWIFT_VERSION is somehow causing things to use the old Swift runtime, or a mixed and incompatible environment?

@compnerd and @buttaface for visibility as they were also working on alignment of types for Android/Windows.

[finagolfin]

Could it be that lowering SWIFT_VERSION is somehow causing things to use the old Swift runtime, or a mixed and incompatible environment?

@grynspan, your change was never merged back to Swift 5.6, which is what he is testing.

@colemancda, you are seeing this with all Swift executables running on Debian armhf?

This could be related to another metadata crash I reported in a NIO test on Android armv7, #58307, though most Android armv7 executables run fine for me.

[colemancda]

No, the 5.5 runtime won't compile with 5.6 compiler and vice versa. That AtomicWaitQueue type is new to the 5.6 runtime. I also wanna note that if I compile the 5.6 Stdlib in release mode and run with qemu-arm-static, it runs and only crashes once every 20 tries.

[grynspan]

@buttaface D'oh! Well, if my change was never merged back, then… this issue is resolved in a future Swift version, I guess?

[finagolfin]

if I compile the 5.6 Stdlib in release mode and run with qemu-arm-static, it runs and only crashes once every 20 tries.

Oh, I missed that you are mostly only seeing this with a Debug build. I haven't built the Debug 5.6 stdlib for armv7, only Release.

Try backporting his pull #42355 to 5.6, maybe that will fix it?

[colemancda]

Adding the patch for #42355 into 5.6.1 release seems to have fixed it. Now I have a crash in the Concurrency module, but that is already filed under another bug.

#58151 (comment)

[colemancda]

Added patches

colemancda/swift-armv7@eb9bf39
https://gist.github.com/colemancda/d88a775f2bf3e234f4fa705c46b66b37

[colemancda]

@grynspan Thanks for the fix!

[grynspan]

@colemancda Happy to help!