Store test content in a custom metadata section.

See also: swiftlang/swift#76698

Resolves #735.

Author: grynspan

Documentation/ABI/TestContent.md

@@ -0,0 +1,112 @@
+# Runtime-discoverable test content
+
+<!--
+This source file is part of the Swift.org open source project
+
+Copyright (c) 2024 Apple Inc. and the Swift project authors
+Licensed under Apache License v2.0 with Runtime Library Exception
+
+See https://swift.org/LICENSE.txt for license information
+See https://swift.org/CONTRIBUTORS.txt for Swift project authors
+-->
+
+This document describes the format and location of test content that the testing
+library emits at compile time and can discover at runtime.
+
+## Basic format
+
+Swift Testing uses the [ELF Note format](https://man7.org/linux/man-pages/man5/elf.5.html)
+to store individual records of test content. Records created and discoverable by
+the testing library are stored in dedicated platform-specific sections:
+
+| Platform | Binary Format | Section Name |
+|-|:-:|-|
+| macOS | Mach-O | `__DATA_CONST,__swift5_tests` |
+| iOS | Mach-O | `__DATA_CONST,__swift5_tests` |
+| watchOS | Mach-O | `__DATA_CONST,__swift5_tests` |
+| tvOS | Mach-O | `__DATA_CONST,__swift5_tests` |
+| visionOS | Mach-O | `__DATA_CONST,__swift5_tests` | 
+| Linux | ELF | `PT_NOTE`[^1] |
+| FreeBSD | ELF | `PT_NOTE`[^1] |
+| Android | ELF | `PT_NOTE`[^1] |
+| WASI | Statically Linked | `swift5_tests` |
+| Windows | PE/COFF | `.sw5test` |
+
+[^1]: On platforms that use the ELF binary format natively, test content records
+      are stored in ELF program headers of type `PT_NOTE`. Take care not to
+      remove these program headers (for example, by invoking [`strip(1)`](https://www.man7.org/linux/man-pages/man1/strip.1.html).)
+
+### Determining the type of test content
+      
+Regardless of platform, all test content records created and discoverable by the
+testing library start have the name `"Swift Testing"` stored in the implied
+`n_name` field of their underlying ELF Notes. Each record's _type_ (stored in
+the underlying ELF Note's `n_type` field) determines how the record will be
+interpreted at runtime:
+
+| Type Value | Interpretation |
+|-:|-|
+| < `0` | Undefined (**do not use**) |
+| `0` ... `99` | Reserved |
+| `100` | Test or suite declaration |
+| `101` | Exit test |
+
+<!-- When adding cases to this enumeration, be sure to also update the
+corresponding enumeration in Discovery.h and TestContentGeneration.swift. -->
+
+### Loading test content from a record
+
+For all currently-defined record types, the header and name are followed by a
+structure of the following form:
+
+```c
+struct SWTTestContent {
+  bool (* accessor)(void *);
+  uint64_t flags;
+};
+```
+
+#### The accessor field
+
+The function `accessor` is a C function whose signature in Swift can be restated
+as:
+
+```swift
+@convention(c) (_ outValue: UnsafeMutableRawPointer) -> Bool
+```
+
+When called, it initializes the memory at `outValue` to an instance of some
+Swift type and returns `true`, or returns `false` if it could not generate the
+relevant content. On successful return, the caller is responsible for
+deinitializing the memory at `outValue` when done with it.
+
+The concrete Swift type of `accessor`'s result depends on the type of record:
+
+| Type Value | Return Type |
+|-:|-|
+| < `0` | Undefined (**do not use**) |
+| `0` ... `99` | `nil` |
+| `100` | `@Sendable () async -> Test`[^2] |
+| `101` | `ExitTest` (owned by caller) |
+
+[^2]: This signature is not the signature of `accessor`, but of the Swift
+      function reference it returns. This level of indirection is necessary
+      because loading a test or suite declaration is an asynchronous operation,
+      but C functions cannot be `async`.
+
+#### The flags field
+
+For test or suite declarations (type `100`), the following flags are defined:
+
+| Bit | Description |
+|-:|-|
+| `1 << 0` | This record contains a suite declaration |
+| `1 << 1` | This record contains a parameterized test function declaration |
+
+For exit test declarations (type `101`), no flags are currently defined.
+
+## Third-party test content
+
+TODO: elaborate how tools can reuse the same `n_name` and `n_type` fields to
+supplement Swift Testing's data, or use a different `n_name` field to store
+arbitrary other data in the test content section that Swift Testing will ignore.

Documentation/Porting.md

@@ -113,13 +113,14 @@ Once the header is included, we can call `GetDateTime()` from `Clock.swift`:
 ## Runtime test discovery
 
 When porting to a new platform, you may need to provide a new implementation for
-`enumerateTypeMetadataSections()` in `Discovery.cpp`. Test discovery is
-dependent on Swift metadata discovery which is an inherently platform-specific
-operation.
+`enumerateTestContentSections()` in `Discovery.cpp`. Test discovery is dependent
+on Swift metadata discovery which is an inherently platform-specific operation.
 
-_Most_ platforms will be able to reuse the implementation used by Linux and
-Windows that calls an internal Swift runtime function to enumerate available
-metadata. If you are porting Swift Testing to Classic, this function won't be
+_Most_ platforms in use today use the ELF image format and will be able to reuse
+the implementation used by Linux. That implementation calls `dl_iterate_phdr()`
+in the GNU C Library to enumerate available metadata.
+
+If you are porting Swift Testing to Classic, `dl_iterate_phdr()` won't be
 available, so you'll need to write a custom implementation instead. Assuming
 that the Swift compiler emits section information into the resource fork on
 Classic, you could use the [Resource Manager](https://developer.apple.com/library/archive/documentation/mac/pdf/MoreMacintoshToolbox.pdf)
@@ -132,16 +133,21 @@ to load that information:
  // ...
 +#elif defined(macintosh)
 +template <typename SectionEnumerator>
-+static void enumerateTypeMetadataSections(const SectionEnumerator& body) {
++static void enumerateTestContentSections(const SectionEnumerator& body) {
 +  ResFileRefNum refNum;
 +  if (noErr == GetTopResourceFile(&refNum)) {
 +    ResFileRefNum oldRefNum = refNum;
 +    do {
 +      UseResFile(refNum);
-+      Handle handle = Get1NamedResource('swft', "\p__swift5_types");
++      Handle handle = Get1NamedResource('swft', "\p__swift5_tests");
 +      if (handle && *handle) {
-+        size_t size = GetHandleSize(handle);
-+        body(*handle, size);
++        auto imageAddress = reinterpret_cast<const void *>(static_cast<uintptr_t>(refNum));
++        SWTSectionBounds sb = { imageAddress, *handle, GetHandleSize(handle) };
++        bool stop = false;
++        body(sb, &stop);
++        if (stop) {
++          break;
++        }
 +      }
 +    } while (noErr == GetNextResourceFile(refNum, &refNum));
 +    UseResFile(oldRefNum);
@@ -150,7 +156,7 @@ to load that information:
  #else
  #warning Platform-specific implementation missing: Runtime test discovery unavailable
  template <typename SectionEnumerator>
- static void enumerateTypeMetadataSections(const SectionEnumerator& body) {}
+ static void enumerateTestContentSections(const SectionEnumerator& body) {}
  #endif
 ```
 

Package.swift

@@ -128,6 +128,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
       .enableExperimentalFeature("AccessLevelOnImport"),
       .enableUpcomingFeature("InternalImportsByDefault"),
 
+      .enableExperimentalFeature("SymbolLinkageMarkers"),
+
       .define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
 
       .define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),

Sources/Testing/ExitTests/ExitTest.swift

@@ -24,15 +24,29 @@ public struct ExitTest: Sendable, ~Copyable {
   /// The expected exit condition of the exit test.
   public var expectedExitCondition: ExitCondition
 
-  /// The body closure of the exit test.
-  fileprivate var body: @Sendable () async throws -> Void = {}
-
   /// The source location of the exit test.
   ///
   /// The source location is unique to each exit test and is consistent between
   /// processes, so it can be used to uniquely identify an exit test at runtime.
   public var sourceLocation: SourceLocation
 
+  /// The body closure of the exit test.
+  fileprivate var body: @Sendable () async throws -> Void
+
+  /// Initialize an exit test at runtime.
+  ///
+  /// - Warning: This initializer is used to implement the `#expect(exitsWith:)`
+  ///   macro. Do not use it directly.
+  public init(
+    __expectedExitCondition expectedExitCondition: ExitCondition,
+    sourceLocation: SourceLocation,
+    body: @escaping @Sendable () async throws -> Void = {}
+  ) {
+    self.expectedExitCondition = expectedExitCondition
+    self.sourceLocation = sourceLocation
+    self.body = body
+  }
+
   /// Disable crash reporting, crash logging, or core dumps for the current
   /// process.
   private static func _disableCrashReporting() {
@@ -102,44 +116,24 @@ public struct ExitTest: Sendable, ~Copyable {
 
 // MARK: - Discovery
 
-/// A protocol describing a type that contains an exit test.
-///
-/// - Warning: This protocol is used to implement the `#expect(exitsWith:)`
-///   macro. Do not use it directly.
-@_alwaysEmitConformanceMetadata
-@_spi(Experimental)
-public protocol __ExitTestContainer {
-  /// The expected exit condition of the exit test.
-  static var __expectedExitCondition: ExitCondition { get }
-
-  /// The source location of the exit test.
-  static var __sourceLocation: SourceLocation { get }
-
-  /// The body function of the exit test.
-  static var __body: @Sendable () async throws -> Void { get }
-}
-
 extension ExitTest {
-  /// A string that appears within all auto-generated types conforming to the
-  /// `__ExitTestContainer` protocol.
-  private static let _exitTestContainerTypeNameMagic = "__🟠$exit_test_body__"
-
   /// Find the exit test function at the given source location.
   ///
   /// - Parameters:
   ///   - sourceLocation: The source location of the exit test to find.
   ///
   /// - Returns: The specified exit test function, or `nil` if no such exit test
   ///   could be found.
+  @_spi(ForToolsIntegrationOnly)
   public static func find(at sourceLocation: SourceLocation) -> Self? {
     var result: Self?
 
-    enumerateTypes(withNamesContaining: _exitTestContainerTypeNameMagic) { _, type, stop in
-      if let type = type as? any __ExitTestContainer.Type, type.__sourceLocation == sourceLocation {
+    enumerateTestContent(ofKind: .exitTest, as: ExitTest.self) { _, exitTest, _, stop in
+      if exitTest.sourceLocation == sourceLocation {
         result = ExitTest(
-          expectedExitCondition: type.__expectedExitCondition,
-          body: type.__body,
-          sourceLocation: type.__sourceLocation
+          __expectedExitCondition: exitTest.expectedExitCondition,
+          sourceLocation: exitTest.sourceLocation,
+          body: exitTest.body
         )
         stop = true
       }
@@ -183,7 +177,7 @@ func callExitTest(
 
   let actualExitCondition: ExitCondition
   do {
-    let exitTest = ExitTest(expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
+    let exitTest = ExitTest(__expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
     actualExitCondition = try await configuration.exitTestHandler(exitTest)
   } catch {
     // An error here would indicate a problem in the exit test handler such as a
@@ -295,7 +289,7 @@ extension ExitTest {
     // External tools authors should set up their own back channel mechanisms
     // and ensure they're installed before calling ExitTest.callAsFunction().
     guard var result = find(at: sourceLocation) else {
-      return nil
+      fatalError("Could not find an exit test that should have been located at \(sourceLocation).")
     }
 
     // We can't say guard let here because it counts as a consume.