Skip to content

[5.9.0][CMake] Support Macros in Linux #68197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Aug 31, 2023

Conversation

rintaro
Copy link
Member

@rintaro rintaro commented Aug 29, 2023

Cherry-pick #68190 into release/5.9.0

  • Explanation: Enable Macros in Linux. Swift parser integration is required for enabling macros. Since Linux doesn't have ABI stable swiftCore in /usr/lib or any other stable location, executables and shared libraries that contains Swift module need to have RUNPATH to the runtime in its own toolchain. This PR forces "hosttools" bootstrapping mode, but set the RUNPATH to the currently-being-built toolchain.
  • Scope: CMake build script
  • Risk: Mid. This PR is carefully crafted not to affect Darwin platforms. For Linux, this is necessary for enabling macros.
  • Testing: Passes current test suite.
  • Issues: rdar://104346187
  • Reviewers: TBA

DougGregor and others added 14 commits August 29, 2023 21:52
When building Swift code into the compiler (e.g, the new Swift parser
along with macros support), make sure we always add the appropriate
paths to (1) link against the host Swift toolchain, and (2) find the
host Swift libraries (such as SwiftSyntax) at runtime.

The CMake code for doing this was only running for Darwin builds, so
generalize it to also work on Linux.

(cherry picked from commit 277619a)
This enables running macro tests on Linux.

(cherry picked from commit c573a7a)
For compiling codes required for macro support, we now need swiftc
compiler in the build machine.

Unlike Darwin OSes, where swiftCore runtime is guaranteed to be present
in /usr/lib, Linux doesn't have ABI stability and the stdlib of the
build machine is not at the specific location. So the built compiler
cannot relies on the shared object in the toolchain.

(cherry picked from commit 9c9010e)
(cherry picked from commit 040d15d)
Many shared libs and executables are only run after stdlib/runtime are
built. They don't need to link with builders stdlib at all.

(cherry picked from commit 2a2787b)
(cherry picked from commit 16bf7a3)
and Swift parser integration is enabled.
If swift parser integration is enabled, SwiftSyntax libraries are always
built with host tools. Other SwiftCompilerSources modules must use the
same runtime with parser libraries.

(cherry picked from commit 0f0c492)
(cherry picked from commit bf0081a)
(cherry picked from commit 9017ef5)
(cherry picked from commit 79f6bca)
SourceKit components are never built while bootstrapping stages. So it
doen't need to have stage 0 or 1 handling.

(cherry picked from commit 914a6fd)
(cherry picked from commit 42360db)
`$ORIGIN` is only supported in Linux, BSD, etc. (i.e. not in Windows)

(cherry picked from commit 602604b)
(cherry picked from commit 64c5430)
In Linux. Instead of setting temporary "fallback" RUNPATH, Set
LD_LIBRARY_PATH to builder's runtime when building standard library.
So we don't need to strip the temporary RUNPATH when installing.

(cherry picked from commit 757aaa3)
(cherry picked from commit f5445f4)
recursive-lipo requires shared libraries have +x permissions.
Use PROGRAMS instead of FILES when installing swift-syntax libraries.

(cherry picked from commit 319b8bd)
(cherry picked from commit 5bbab90)
libSwiftScan is built in 'lib' but installed in 'lib/swift/host' RUNPATH
should have correct '$ORIGIN/../{platform}' to load 'swiftCore' runtime
library.

(cherry picked from commit 4ccc866)
(cherry picked from commit 3badb3b)
swift-compatibility-symbols, swift-def-to-strings-converter,
and swift-serialize-diagnostics don't use any Swift modules. But when
SWIFT_SWIFT_PARSER was enabled, they are linked with swiftCore. But
these binaries can be executed before the runtime is being built.
We need to stop them linking with swiftCore.

(cherry picked from commit dc68773)
(cherry picked from commit 1e7c84a)
RPATH_SET is not available until cmake 3.21.0. Use RPATH_CHANGE instead.

(cherry picked from commit f645069)
(cherry picked from commit c39f95b)
@rintaro rintaro requested a review from a team as a code owner August 29, 2023 22:07
@rintaro
Copy link
Member Author

rintaro commented Aug 29, 2023

@rintaro
Copy link
Member Author

rintaro commented Aug 29, 2023

1 similar comment
@rintaro
Copy link
Member Author

rintaro commented Aug 29, 2023

@shahmishal
Copy link
Member

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

@shahmishal
Copy link
Member

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

@shahmishal
Copy link
Member

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

@shahmishal
Copy link
Member

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

@shahmishal
Copy link
Member

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

@shahmishal
Copy link
Member

=== /home/build-user/swift/stdlib/public/core/Macros.swift:34 ===
Traceback (most recent call last):
  File "/home/build-user/swift/utils/line-directive", line 738, in <module>
    run()
  File "/home/build-user/swift/utils/line-directive", line 732, in run
    sys.stdout.write(output)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2506' in position 10: ordinal not in range(128)
ninja: build stopped: subcommand failed.

Ubuntu 18.04 failure

rintaro and others added 5 commits August 30, 2023 15:38
Some non-stdlib thing e.g. swift-backtrace still might be built before
the runtime is built. For building Swift code in Linux "hosttools",
always set 'LD_LIBRARY_PATH' to the runtime in the builder.

(cherry picked from commit 0aa0aac)
(cherry picked from commit d8a1265)
RPATH to 'lib/swift/host' was missing in Darwin platform.

(cherry picked from commit 9d78971)
(cherry picked from commit c244dfc)
"Support Macros in Linux" patches was inconsistent with checking
platforms. Some only checked 'LINUX' but some matches
'LINUX|ANDROID|OPENBSD|FREEBSD'. Although I don't have tested other
platoforms than Linux at all, there's no reason to limit it to Linux.
So use the consistent check to match 'LINUX|ANDROID|OPENBSD|FREEBSD'

(cherry picked from commit 8049922)
(cherry picked from commit 714589e)
When Swift parser integration is disabled (i.e. NOT SWIFT_SWIFT_PARSER),
macro plugins targets are not created. So CMake should bail out.

(cherry picked from commit ad2dacd)
(cherry picked from commit 5125ac3)
Ubuntu 18.04 seems to be defaulting to 'ascii', causing an error when
using line-directive combined with the utf-8 diagnostic output used when
early swift syntax is enabled.

(cherry picked from commit afd3f17)
(cherry picked from commit f038280)
@rintaro rintaro force-pushed the 5.9.0-swift-swift-linux branch from c653d2d to 5ba02ab Compare August 30, 2023 22:40
@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

@rintaro rintaro force-pushed the 5.9.0-swift-swift-linux branch from 5ba02ab to 82ab9b7 Compare August 30, 2023 23:18
@rintaro rintaro force-pushed the 5.9.0-swift-swift-linux branch from 82ab9b7 to d34a60d Compare August 30, 2023 23:36
@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain CentOS 7

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain UBI9

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 18.04

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Ubuntu 22.04

@rintaro
Copy link
Member Author

rintaro commented Aug 30, 2023

swiftlang/llvm-project#7354
swiftlang/swift-syntax#2117
@swift-ci Please Build Toolchain Amazon Linux 2

@shahmishal shahmishal merged commit f3891f2 into swiftlang:release/5.9.0 Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants