[5.9.0][CMake] Support Macros in Linux #68197

rintaro · 2023-08-29T22:07:09Z

Cherry-pick #68190 into release/5.9.0

Explanation: Enable Macros in Linux. Swift parser integration is required for enabling macros. Since Linux doesn't have ABI stable swiftCore in /usr/lib or any other stable location, executables and shared libraries that contains Swift module need to have RUNPATH to the runtime in its own toolchain. This PR forces "hosttools" bootstrapping mode, but set the RUNPATH to the currently-being-built toolchain.
Scope: CMake build script
Risk: Mid. This PR is carefully crafted not to affect Darwin platforms. For Linux, this is necessary for enabling macros.
Testing: Passes current test suite.
Issues: rdar://104346187
Reviewers: TBA

When building Swift code into the compiler (e.g, the new Swift parser along with macros support), make sure we always add the appropriate paths to (1) link against the host Swift toolchain, and (2) find the host Swift libraries (such as SwiftSyntax) at runtime. The CMake code for doing this was only running for Darwin builds, so generalize it to also work on Linux. (cherry picked from commit 277619a)

This enables running macro tests on Linux. (cherry picked from commit c573a7a)

(cherry picked from commit defe9eb)

For compiling codes required for macro support, we now need swiftc compiler in the build machine. Unlike Darwin OSes, where swiftCore runtime is guaranteed to be present in /usr/lib, Linux doesn't have ABI stability and the stdlib of the build machine is not at the specific location. So the built compiler cannot relies on the shared object in the toolchain. (cherry picked from commit 9c9010e) (cherry picked from commit 040d15d)

Many shared libs and executables are only run after stdlib/runtime are built. They don't need to link with builders stdlib at all. (cherry picked from commit 2a2787b) (cherry picked from commit 16bf7a3)

and Swift parser integration is enabled. If swift parser integration is enabled, SwiftSyntax libraries are always built with host tools. Other SwiftCompilerSources modules must use the same runtime with parser libraries. (cherry picked from commit 0f0c492) (cherry picked from commit bf0081a)

(cherry picked from commit 9017ef5) (cherry picked from commit 79f6bca)

SourceKit components are never built while bootstrapping stages. So it doen't need to have stage 0 or 1 handling. (cherry picked from commit 914a6fd) (cherry picked from commit 42360db)

`$ORIGIN` is only supported in Linux, BSD, etc. (i.e. not in Windows) (cherry picked from commit 602604b) (cherry picked from commit 64c5430)

In Linux. Instead of setting temporary "fallback" RUNPATH, Set LD_LIBRARY_PATH to builder's runtime when building standard library. So we don't need to strip the temporary RUNPATH when installing. (cherry picked from commit 757aaa3) (cherry picked from commit f5445f4)

recursive-lipo requires shared libraries have +x permissions. Use PROGRAMS instead of FILES when installing swift-syntax libraries. (cherry picked from commit 319b8bd) (cherry picked from commit 5bbab90)

libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime library. (cherry picked from commit 4ccc866) (cherry picked from commit 3badb3b)

swift-compatibility-symbols, swift-def-to-strings-converter, and swift-serialize-diagnostics don't use any Swift modules. But when SWIFT_SWIFT_PARSER was enabled, they are linked with swiftCore. But these binaries can be executed before the runtime is being built. We need to stop them linking with swiftCore. (cherry picked from commit dc68773) (cherry picked from commit 1e7c84a)

RPATH_SET is not available until cmake 3.21.0. Use RPATH_CHANGE instead. (cherry picked from commit f645069) (cherry picked from commit c39f95b)

rintaro · 2023-08-29T22:07:45Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please test

rintaro · 2023-08-29T22:07:53Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain

rintaro · 2023-08-29T22:10:06Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain

shahmishal · 2023-08-30T00:43:52Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

shahmishal · 2023-08-30T00:44:16Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

shahmishal · 2023-08-30T00:44:45Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

shahmishal · 2023-08-30T00:45:03Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

shahmishal · 2023-08-30T00:45:17Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

shahmishal · 2023-08-30T04:08:29Z

=== /home/build-user/swift/stdlib/public/core/Macros.swift:34 ===
Traceback (most recent call last):
  File "/home/build-user/swift/utils/line-directive", line 738, in <module>
    run()
  File "/home/build-user/swift/utils/line-directive", line 732, in run
    sys.stdout.write(output)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2506' in position 10: ordinal not in range(128)
ninja: build stopped: subcommand failed.

Ubuntu 18.04 failure

Some non-stdlib thing e.g. swift-backtrace still might be built before the runtime is built. For building Swift code in Linux "hosttools", always set 'LD_LIBRARY_PATH' to the runtime in the builder. (cherry picked from commit 0aa0aac) (cherry picked from commit d8a1265)

RPATH to 'lib/swift/host' was missing in Darwin platform. (cherry picked from commit 9d78971) (cherry picked from commit c244dfc)

"Support Macros in Linux" patches was inconsistent with checking platforms. Some only checked 'LINUX' but some matches 'LINUX|ANDROID|OPENBSD|FREEBSD'. Although I don't have tested other platoforms than Linux at all, there's no reason to limit it to Linux. So use the consistent check to match 'LINUX|ANDROID|OPENBSD|FREEBSD' (cherry picked from commit 8049922) (cherry picked from commit 714589e)

When Swift parser integration is disabled (i.e. NOT SWIFT_SWIFT_PARSER), macro plugins targets are not created. So CMake should bail out. (cherry picked from commit ad2dacd) (cherry picked from commit 5125ac3)

Ubuntu 18.04 seems to be defaulting to 'ascii', causing an error when using line-directive combined with the utf-8 diagnostic output used when early swift syntax is enabled. (cherry picked from commit afd3f17) (cherry picked from commit f038280)

rintaro · 2023-08-30T22:41:18Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please test

rintaro · 2023-08-30T22:41:25Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain

rintaro · 2023-08-30T22:42:27Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

rintaro · 2023-08-30T22:42:45Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

rintaro · 2023-08-30T22:42:55Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

rintaro · 2023-08-30T22:43:07Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

rintaro · 2023-08-30T22:43:21Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

(cherry picked from commit 71f5b47)

rintaro · 2023-08-30T23:37:54Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please test

rintaro · 2023-08-30T23:38:53Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

rintaro · 2023-08-30T23:39:09Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain

rintaro · 2023-08-30T23:39:15Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

rintaro · 2023-08-30T23:39:26Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

rintaro · 2023-08-30T23:39:44Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

rintaro · 2023-08-30T23:39:52Z

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

DougGregor and others added 14 commits August 29, 2023 21:52

[CMake] Fix host library builds and rpaths for testing macros

4690e59

This enables running macro tests on Linux. (cherry picked from commit c573a7a)

[SourceKit] Fix paths when including the new Swift compiler

b3b1b7a

(cherry picked from commit defe9eb)

Avoid adding redundant temporary RUNPATH to builder's stdlib

e5e63db

Many shared libs and executables are only run after stdlib/runtime are built. They don't need to link with builders stdlib at all. (cherry picked from commit 2a2787b) (cherry picked from commit 16bf7a3)

[CMake] Update for review comments

046a144

(cherry picked from commit 9017ef5) (cherry picked from commit 79f6bca)

[CMake] Fix weiredness in swift runtime linking for SourceKit libraries

ed2b5df

SourceKit components are never built while bootstrapping stages. So it doen't need to have stage 0 or 1 handling. (cherry picked from commit 914a6fd) (cherry picked from commit 42360db)

[CMake] Further cleanup for RUNPATH setting

a5d05ea

`$ORIGIN` is only supported in Linux, BSD, etc. (i.e. not in Windows) (cherry picked from commit 602604b) (cherry picked from commit 64c5430)

[CMake] Fix install for macOS toolchain

3c79ce5

recursive-lipo requires shared libraries have +x permissions. Use PROGRAMS instead of FILES when installing swift-syntax libraries. (cherry picked from commit 319b8bd) (cherry picked from commit 5bbab90)

[CMake] Fix RUNPATH of lib_InternalSwiftScan in Linux

43c4e2f

libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime library. (cherry picked from commit 4ccc866) (cherry picked from commit 3badb3b)

[CMake] Use RPATH_CHANGE instead RPATH_SET

56d0e65

RPATH_SET is not available until cmake 3.21.0. Use RPATH_CHANGE instead. (cherry picked from commit f645069) (cherry picked from commit c39f95b)

rintaro requested a review from a team as a code owner August 29, 2023 22:07

rintaro mentioned this pull request Aug 29, 2023

[5.9.0][CMake] Add padding to RUNPATH in linux swiftlang/swift-syntax#2117

Merged

rintaro and others added 5 commits August 30, 2023 15:38

[CMake][SourceKit] Add RPATH to 'lib/swift/host'

704987e

RPATH to 'lib/swift/host' was missing in Darwin platform. (cherry picked from commit 9d78971) (cherry picked from commit c244dfc)

[CMake] Handle cases where Swift parser integration is disabled in Linux

78188b3

When Swift parser integration is disabled (i.e. NOT SWIFT_SWIFT_PARSER), macro plugins targets are not created. So CMake should bail out. (cherry picked from commit ad2dacd) (cherry picked from commit 5125ac3)

rintaro force-pushed the 5.9.0-swift-swift-linux branch from c653d2d to 5ba02ab Compare August 30, 2023 22:40

rintaro force-pushed the 5.9.0-swift-swift-linux branch from 5ba02ab to 82ab9b7 Compare August 30, 2023 23:18

[CMake] Instantiate missing C++ templates

d34a60d

(cherry picked from commit 71f5b47)

rintaro force-pushed the 5.9.0-swift-swift-linux branch from 82ab9b7 to d34a60d Compare August 30, 2023 23:36

airspeedswift approved these changes Aug 31, 2023

View reviewed changes

shahmishal merged commit f3891f2 into swiftlang:release/5.9.0 Aug 31, 2023

[5.9.0][CMake] Support Macros in Linux #68197

[5.9.0][CMake] Support Macros in Linux #68197

Uh oh!

Conversation

rintaro commented Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

rintaro commented Aug 29, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

shahmishal commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

rintaro commented Aug 30, 2023

Uh oh!

Uh oh!

rintaro commented Aug 29, 2023 •

edited

Loading