[SR-3998] SourceKit build failure on Linux without blocks runtime

[swift-ci]

Previous ID	SR-3998
Radar	rdar://problem/33488928
Original Reporter	Vladislav (JIRA User)
Type	Bug
Status	Resolved
Resolution	Done

Environment

Linux ved-hp-gentoo 4.9.6-gentoo-r1 #5 SMP Sun Feb 5 21:47:04 MSK 2017 x86_64 Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz GenuineIntel GNU/Linux

Additional Detail from JIRA

Votes	2
Component/s	Source Tooling
Labels	Bug, BuildScript, Linux
Assignee	None
Priority	Medium

md5: b61152b9d429beca7da1f76fc3589670

Issue Description:

I did fresh checkout and run standarad bild script:

utils/build-script --preset=buildbot_linux install_destdir=/mnt/swift/gentoo installable_package=/mnt/swift/gentoo/swift-gentoo.tar.gz
During execution I have got the following error:
[545/773] Building CXX object tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o
FAILED: tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o
/usr/bin/clang++ -DCMARK_STATIC_DEFINE -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/SourceKit/lib/Support -I/mnt/swift/swift-source/swift/tools/SourceKit/lib/Support -I/mnt/swift/swift-source/swift/tools/SourceKit/include -Iinclude -I/mnt/swift/swift-source/swift/include -I/mnt/swift/swift-source/llvm/include -I/mnt/swift/swift-source/build/buildbot_linux/llvm-linux-x86_64/include -I/mnt/swift/swift-source/build/buildbot_linux/llvm-linux-x86_64/tools/clang/include -I/mnt/swift/swift-source/llvm/tools/clang/include -I/mnt/swift/swift-source/cmark/src -I/mnt/swift/swift-source/build/buildbot_linux/cmark-linux-x86_64/src -I/mnt/swift/swift-source/swift-corelibs-libdispatch -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -Werror=date-time -std=c++11 -fcolor-diagnostics -ffunction-sections -fdata-sections -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -O3 -UNDEBUG -fno-exceptions -fno-rtti -target x86_64-unknown-linux-gnu -O2 -momit-leaf-frame-pointer -g0 -UNDEBUG -fblocks -MD -MT tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o -MF tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o.d -o tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o -c /mnt/swift/swift-source/swift/tools/SourceKit/lib/Support/Concurrency-libdispatch.cpp
/mnt/swift/swift-source/swift/tools/SourceKit/lib/Support/Concurrency-libdispatch.cpp:20:10: fatal error: 'Block.h' file not found
#include <Block.h>
^
1 error generated.
Is it possible to modify cmake scripts to include -I/.../swift-source/compiler-rt/lib/BlocksRuntime?

[swift-ci]

Comment by Hugh Bellamy (JIRA)

For now you can work around this by using the CMake -DSWIFT_BUILD_SOURCEKIT=FALSE

(https://github.com/apple/swift/blob/master/CMakeLists.txt#L373)

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Thank you Hugh.
It is not clear how to pass this cmake option to the standard script:

utils/build-script --preset=buildbot_linux install_destdir=/mnt/swift/gentoo installable_package=/mnt/swift/gentoo/swift-gentoo.tar.gz

I have find another workaround but I am not sure if it is correct.
I have build and install BlocksRuntime library according to the instructions given at https://github.com/mackyle/blocksruntime
I just changed the buildlib script adding -fPIC options to the compilation commands:

$CC" -c $CFLAGS -fPIC -o $SRC/data.o $SRC/data.c \
&& "$CC" -c $CFLAGS -fPIC -o $SRC/runtime.o -I . $SRC/runtime.c

Currently, it looks like libdispatch is building normally.

[alblue]

The build of SourceKit shells out to libdispatch via a Makefile, rather than CMake. We have multiple copies of the blocks runtime in the project at the moment. It may make sense for SourceKit to depend on dispatch's copy but this would run into problems if (for example) clients of SourceKitd want to use blocks. There's probably a bigger issue to solve here.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Thank you, Alex!
So, if I understood correctly you and Hugh do not recommend to build SourceKit on Linux at all. Am I right?
If yes, how to pass -DSWIFT_BUILD_SOURCEKIT=FALSE to the utils/build-script --preset=buildbot_linux?

[alblue]

"On Linux distributions without the blocks runtime installed" - it builds fine on the Ubuntu images used by the cI bot 🙂

That said, it's a problem that needs to be fixed. In the interim you can build without sourcekit. If you can modify your local copy of the sources, you can change the default SWIFT_BUILD_SOURCEKIT_default here:

https://github.com/apple/swift/blob/c440db63e981c32d66ab10269950d7769f00642b/CMakeLists.txt#L366

If you want to pass it in the build script then there's a --extra-cmake-options option which you can use, so something like:

utils/build-script ---preset=buildbot_linux --extra-cmake-options=-DSWIFT_BUILD_SOURCEKIT=false

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Thank you Alex! I have just did fresh checkout. But therу is something wrong with build_script. It does not accepted option ---extra-cmake-options

utils/build-script --preset=buildbot_linux --extra-cmake-options=-DSWIFT_BUILD_SOURCEKIT=false install_destdir=/mnt/swift/gentoo installable_package=/mnt/swift/gentoo/swift-gentoo.tar.gz
usage: build-script [h] [n] [preset-file PATH] [-preset NAME]
[show-presets [BOOL]] [-distcc [BOOL]] [-j BUILD_JOBS]
[--expand-build-script-invocation [BOOL]]
[SUBSTITUTION [SUBSTITUTION ...]]
build-script: error: unrecognized arguments: --extra-cmake-options=-DSWIFT_BUILD_SOURCEKIT=false

And help says that it should not accept any options if using presets.
I did edirecet edit of CMakeLists.txt and started build. Will wait for finish.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

No. On Gentoo build process stops with error I have alrady reported at SR-4074
Tomorrow wil try to repeat on Linux From Scratch.

[alblue]

If you run the build-script with a preset, you can use --expand-build-script-invocation to see what it would have run via the build script itself (in other words, expand the presets). You could use that to then add the extra-cmake-options argument.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Thank you, Alex! Therefore, I did fresh checkout, expand preset and run following build script o Linux From Scratch:

utils/build-script --assertions --no-swift-stdlib-assertions --llbuild --swiftpm --xctest --build-subdir=buildbot_linux --lldb --release --extra-cmake-options=-DSWIFT_BUILD_SOURCEKIT=false --foundation --libdispatch --lit-args=v - --swift-enable-ast-verifier=0 --build-ninja --install-swift --install-lldb --install-llbuild --install-swiftpm --install-xctest --install-prefix=/usr '--swift-install-components=autolink-driver;compiler;clang-builtin-headers;stdlib;swift-remote-mirror;sdk-overlay;license' --build-swift-static-stdlib --build-swift-static-sdk-overlay --build-swift-stdlib-unittest-extra --install-destdir=/mnt/swift/gentoo --installable-package=/mnt/swift/gentoo/swift-gentoo.tar.gz --install-foundation --install-libdispatch --reconfigure
As you can see I inserted --extra-cmake-options instead of tests just to save some time. Nevertheless, build process has been aborted at CoreFondation phase with the following error:
CoreFoundation/Collections.subproj/CFBasicHash.c:14:10: fatal error: 'Block.h' file not found
Sorry, quote style strikes some lines.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Today I did fresh checkout but error is the same.
Build script:

utils/build-script --no-assertions --no-swift-stdlib-assertions --llbuild --swiftpm --xctest --build-subdir=buildbot_linux --lldb --release --extra-cmake-options="-DSWIFT_BUILD_SOURCEKIT=false -DLLVM_TARGETS_TO_BUILD=X86" --foundation --libdispatch --lit-args=v - --swift-enable-ast-verifier=0 --build-ninja --install-swift --install-lldb --install-llbuild --install-swiftpm --install-xctest --install-prefix=/usr '--swift-install-components=autolink-driver;compiler;clang-builtin-headers;stdlib;swift-remote-mirror;sdk-overlay;license' --build-swift-static-stdlib --build-swift-static-sdk-overlay --install-destdir=/mnt/swift/lfs --installable-package=/mnt/swift/lfs/swift-lfs.tar.gz --install-foundation --install-libdispatch --reconfigure
Error report:
[109/320] CompileC: CoreFoundation/Collections.subproj/CFBasicHash.c
FAILED: ../build/buildbot_linux/foundation-linux-x86_64/Foundation/CoreFoundation/Collections.subproj/CFBasicHash.c.o
mkdir -p dirname ../build/buildbot_linux/foundation-linux-x86_64/Foundation/CoreFoundation/Collections.subproj/CFBasicHash.c.o; /mnt/swift/swift-source/build/buildbot_linux/llvm-linux-x86_64/bin/clang -fcolor-diagnostics -fdollars-in-identifiers -fblocks -fobjc-runtime=macosx-10.11 -fintegrated-as -fPIC --target=x86_64-linux-gnu -O2 -Ibootstrap/common/usr/include -Ibootstrap/common/usr/local/include -Ibootstrap/x86_64-linux-gnu/usr/include -Ibootstrap/x86_64-linux-gnu/usr/local/include -O2 -g -march=native -pipe -I../build/buildbot_linux/foundation-linux-x86_64/Foundation -I../build/buildbot_linux/foundation-linux-x86_64 -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -I../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation -DDEPLOYMENT_TARGET_LINUX -D_GNU_SOURCE -DCF_CHARACTERSET_DATA_DIR="CoreFoundation/CharacterSets"-DU_SHOW_DRAFT_API -DCF_BUILDING_CF -DDEPLOYMENT_RUNTIME_SWIFT -fconstant-cfstrings -fexceptions -Wno-shorten-64-to-32 -Wno-deprecated-declarations -Wno-unreachable-code -Wno-conditional-uninitialized -Wno-unused-variable -Wno-int-conversion -Wno-unused-function -I/usr/include/libxml2 -I/usr/include/curl -I./ -DDEPLOYMENT_ENABLE_LIBDISPATCH -I/mnt/swift/swift-source/swift-corelibs-libdispatch -I/mnt/swift/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/tests -include CoreFoundation/Base.subproj/CoreFoundation_Prefix.h -c CoreFoundation/Collections.subproj/CFBasicHash.c -o ../build/buildbot_linux/foundation-linux-x86_64/Foundation/CoreFoundation/Collections.subproj/CFBasicHash.c.o
In file included from CoreFoundation/Collections.subproj/CFBasicHash.c:11:
In file included from CoreFoundation/Collections.subproj/CFBasicHash.h:12:
In file included from ../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation/CFInternal.h:138:
../build/buildbot_linux/foundation-linux-x86_64/Foundation/usr//lib/swift/CoreFoundation/ForFoundationOnly.h:113:160: warning: array parameter is missing a nullability type specifier (_Nonnull, _Nullable, or _Null_unspecified) [-Wnullability-completeness-on-arrays]
void (*_Null_unspecified getKeysAndValues)(CFAllocatorRef _Nullable alloc, CFTypeRef context, void *domain, void *_Null_unspecified * _Null_unspecified buf[], CFIndex *numKeyValuePairs);
many warnings and
CoreFoundation/Collections.subproj/CFBasicHash.c:14:10: fatal error: 'Block.h' file not found
#include <Block.h>
^~~~~~~~~
18 warnings and 1 error generated.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Today I did fresh checkout and have got the same error on LFS.

CoreFoundation/Collections.subproj/CFBasicHash.c:14:10: fatal error: 'Block.h' file not found
#include <Block.h>
^~~~~~~~~
18 warnings and 1 error generated.

Build script was:
utils/build-script --extra-cmake-options="-DSWIFT_BUILD_SOURCEKIT=false -DLLVM_TARGETS_TO_BUILD=X86" --no-assertions --no-swift-stdlib-assertions --llbuild --swiftpm --xctest --build-subdir=buildbot_linux --lldb --release --foundation --libdispatch --lit-args=v - --swift-enable-ast-verifier=0 --build-ninja --install-swift --install-lldb --install-llbuild --install-swiftpm --install-xctest --install-prefix=/usr '--swift-install-components=autolink-driver;compiler;clang-builtin-headers;stdlib;swift-remote-mirror;sdk-overlay;license' --build-swift-static-stdlib --build-swift-static-sdk-overlay --install-destdir=/mnt/swift/lfs --installable-package=/mnt/swift/lfs/swift-lfs.tar.gz --install-foundation --install-libdispatch --reconfigure

[alblue]

No need to keep pasting the same message. If you don't have the blocks runtime installed, you'll continue to get this message.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

I did install libblocksruntime from Ubuntu sources and curtly I do not see that issue

[ahti]

This issue still occurs, and I'm not aware of any word from maintainers that this is not a use case they want to support eventually (iirc, Dispatch already moved from requiring a blocks runtime present in the system to using a bundled blocks runtime).

FWIW, Arch Linux is also on the list of distributions without a blocks runtime package.

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Today I have updated Gentoo and did fresh Swift checkout. Nevetheless, I have got the similar error:

[553/835] Building CXX object tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o
FAILED: tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o
/usr/bin/clang++ -DCMARK_STATIC_DEFINE -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/SourceKit/lib/Support -I/mnt/swift/swift-source/swift/tools/SourceKit/lib/Support -I/mnt/swift/swift-source/swift/tools/SourceKit/include -Iinclude -I/mnt/swift/swift-source/swift/include -I/mnt/swift/swift-source/llvm/include -I/mnt/swift/swift-source/build/buildbot_linux/llvm-linux-x86_64/include -I/mnt/swift/swift-source/build/buildbot_linux/llvm-linux-x86_64/tools/clang/include -I/mnt/swift/swift-source/llvm/tools/clang/include -I/mnt/swift/swift-source/cmark/src -I/mnt/swift/swift-source/build/buildbot_linux/cmark-linux-x86_64/src -I/mnt/swift/swift-source/swift-corelibs-libdispatch -fno-stack-protector -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -Werror=date-time -std=c++11 -fcolor-diagnostics -ffunction-sections -fdata-sections -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O3 -UNDEBUG -fno-exceptions -fno-rtti -target x86_64-unknown-linux-gnu -O2 -g0 -UNDEBUG -fblocks -MD -MT tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o -MF tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o.d -o tools/SourceKit/lib/Support/CMakeFiles/SourceKitSupport.dir/Concurrency-libdispatch.cpp.o -c /mnt/swift/swift-source/swift/tools/SourceKit/lib/Support/Concurrency-libdispatch.cpp
/mnt/swift/swift-source/swift/tools/SourceKit/lib/Support/Concurrency-libdispatch.cpp:20:10: fatal error: 'Block.h' file not found
#include <Block.h>
^
1 error generated.

[kevints]

The same error Vladislav (JIRA User)] reported above occurs when building on Ubuntu 14.04 LTS as well unless libblocksruntime-dev is installed. The system package manager installs an ancient version from 2010, libblocksruntime-0.1. This seems odd especially since the current sources for libblocksruntime are already in our tree under compiler-rt (though they appear to be excluded from the cmake config). Perhaps we just need to add CMakeLists for them?

@swift-ci create

[swift-ci]

Comment by Vladislav Dembskiy (JIRA)

Yes, I think it would be good if libblocksruntime will be build on every system there it does not exists. And compiler-rt probably is the best source for it.

[compnerd]

We build blocks runtime and share it across everything now.