Skip to content

Android: add bootstrap build support and fix tests #2417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 9, 2019
Merged

Android: add bootstrap build support and fix tests #2417

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Sources/PackageLoading/Target+PkgConfig.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,9 @@ extension SystemPackageProviderDescription {
if case .linux(.debian) = platform {
return true
}
if case .android = platform {
return true
}
}
return false
}
Expand Down
2 changes: 1 addition & 1 deletion Tests/PackageLoadingTests/PackageBuilderTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ class PackageBuilderTests: XCTestCase {

#if os(Linux)
result.checkDiagnostic("ignoring target 'MyPackageTests' in package 'MyPackage'; C language in tests is not yet supported")
#elseif os(macOS)
#elseif os(macOS) || os(Android)
result.checkProduct("MyPackagePackageTests") { _ in }
#endif
}
Expand Down
4 changes: 4 additions & 0 deletions Tests/TSCBasicTests/FileSystemTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,11 @@ class FileSystemTests: XCTestCase {

// Check read/write against root.
XCTAssertThrows(FileSystemError.ioError) {
#if os(Android)
_ = try fs.readFileContents(AbsolutePath("/system/"))
#else
_ = try fs.readFileContents(AbsolutePath("/"))
#endif
}
XCTAssertThrows(FileSystemError.isDirectory) {
try fs.writeFileContents(AbsolutePath("/"), bytes: [])
Expand Down
16 changes: 15 additions & 1 deletion Tests/TSCBasicTests/PathShimTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,16 +64,30 @@ class PathShimTests : XCTestCase {
class WalkTests : XCTestCase {

func testNonRecursive() {
#if os(Android)
let root = "/system"
var expected = [
AbsolutePath("\(root)/usr"),
AbsolutePath("\(root)/bin"),
AbsolutePath("\(root)/xbin")
]
#else
let root = ""
var expected = [
AbsolutePath("/usr"),
AbsolutePath("/bin"),
AbsolutePath("/sbin")
]
for x in try! walk(AbsolutePath("/"), recursively: false) {
#endif
for x in try! walk(AbsolutePath("\(root)/"), recursively: false) {
if let i = expected.firstIndex(of: x) {
expected.remove(at: i)
}
#if os(Android)
XCTAssertEqual(3, x.components.count)
#else
XCTAssertEqual(2, x.components.count)
#endif
}
XCTAssertEqual(expected.count, 0)
}
Expand Down
19 changes: 17 additions & 2 deletions Utilities/bootstrap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,10 @@ class Target(object):
if args.llbuild_link_framework:
other_args.extend(["-F", args.llbuild_build_dir])

# Don't use GNU strerror_r on Android.
if 'ANDROID_DATA' in os.environ:
other_args.extend(["-Xcc", "-U_GNU_SOURCE"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to move SwiftPM away from this bootstrap script and rely the CMake 3.15.1 which has builtin Swift support. I don't know when that'll happen but I expect to it to happen within next couple of months (ofc not a guarantee). So this might mean you have to do some of these changes again. If you want to try the new WIP CMake bootstrapping, run Utilities/new-bootstrap.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I'm going to try cross-compiling SPM itself from linux to Android next, so I was going to try the CMake support for that.


print(" import-paths: %s" % json.dumps(import_paths), file=output)
print(" other-args: %s" % json.dumps(other_args),
file=output)
Expand Down Expand Up @@ -681,7 +685,7 @@ def process_runtime_libraries(build, args, lib_path):
cmd = [args.swiftc_path, "-emit-library", "-o", runtime_lib_path,
"-Xlinker", "--whole-archive",
"-Xlinker", input_lib_path,
"-Xlinker", "--no-whole-archive", "-lswiftGlibc",
"-Xlinker", "--no-whole-archive", "-lswiftGlibc", "-lFoundation",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only part of this pull that affects non-Android platforms. I had issues for a long time with getting the bootstrapped SPM to interpret its own manifest and dump it to JSON (as mentioned in #2396) and it turned out this was why.

The problem is that libPackageDescription.so depends on some functions from Foundation, but doesn't list it as a required dependency:

> readelf -sW 512-release/usr/lib/swift/pm/4_2/libPackageDescription.so | ag Foundation
   160: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $s10Foundation11JSONEncoderCACycfc
   161: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $s10Foundation11JSONEncoderCMa
   162: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $sSS10FoundationE4data8encodingSSSgAA4DataVh_SSAAE8EncodingVtcfC
   163: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $sSS10FoundationE8EncodingV4utf8ACvau
  1616: 000000000001d130    78 FUNC    LOCAL  HIDDEN    11 $s10Foundation4DataV15_RepresentationOWOe
  2341: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $s10Foundation11JSONEncoderCACycfc
  2342: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $s10Foundation11JSONEncoderCMa
  2445: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $sSS10FoundationE4data8encodingSSSgAA4DataVh_SSAAE8EncodingVtcfC
  2446: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND $sSS10FoundationE8EncodingV4utf8ACvau
> readelf -d 512-release/usr/lib/swift/pm/4_2/libPackageDescription.so| ag library
 0x0000000000000001 (NEEDED)             Shared library: [libswiftSwiftOnoneSupport.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftCore.so]
 0x0000000000000001 (NEEDED)             Shared library: [libswiftGlibc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/buildnode/jenkins/workspace/oss-swift-5.1-package-linux-ubuntu-18_04/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux:$ORIGIN/../../linux]

These results were from the official 5.1.2 release for linux x86_64, I get the same results with the latest snapshots from master.

The Android dynamic linker enforces that even if a executable or library links against libPackageDescription.so and lists libFoundation.so as a dependency there, it isn't transitively applied to libPackageDescription.so, unlike the GNU dynamic linker historically, though that may be changing. It appears that's why this hasn't hit on linux, as the Swift compiler seems to automatically add the needed Foundation dependency to a linux executable I tested linking against libPackageDescription.so, but that doesn't help on Android since it enforces that each library list its requirements itself.

"-Xlinker", "-rpath=$ORIGIN/../../linux"]

# We need to pass one swift file here to bypass the "no input files"
Expand Down Expand Up @@ -1024,7 +1028,14 @@ def main():
if platform.system() == 'Darwin':
build_target = "x86_64-apple-macosx"
elif platform.system() == 'Linux':
if platform.machine() == 'x86_64':
if 'ANDROID_DATA' in os.environ:
if platform.machine().startswith("armv7"):
build_target = 'armv7-unknown-linux-androideabi'
elif platform.machine() == 'aarch64':
build_target = 'aarch64-unknown-linux-android'
else:
raise SystemExit("ERROR: unsupported Android platform:",platform.machine())
elif platform.machine() == 'x86_64':
build_target = "x86_64-unknown-linux-gnu"
elif platform.machine() == "i686":
build_target = "i686-unknown-linux"
Expand Down Expand Up @@ -1175,6 +1186,10 @@ def main():
# Append the versioning build flags.
build_flags.extend(create_versoning_args(args))

# Don't use GNU strerror_r on Android.
if 'ANDROID_DATA' in os.environ:
build_flags.extend(["-Xswiftc", "-Xcc", "-Xswiftc", "-U_GNU_SOURCE"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now but this should go to BuildPlan.swift so swiftpm can correct compile itself and other Swift targets for Android.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, _GNU_SOURCE was set for all clang imports for Android in the Swift compiler a year ago, swiftlang/swift@60e2205, so changing that for all packages built for Android could break stuff. This tweak simply unsets it so the GNU strerror_r isn't used when building SPM itself, and avoids those larger issues.


# Add llbuild import flags.
for import_path in llbuild_import_paths(args):
build_flags.extend(["-Xswiftc", "-I%s" % import_path])
Expand Down