Skip to content

Commit c21b466

Browse files
committed
Store test content in a custom metadata section.
See also: swiftlang/swift#76698 Resolves #735.
1 parent 5763213 commit c21b466

17 files changed

+646
-430
lines changed

Documentation/ABI/TestContent.md

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
# Runtime-discoverable test content
2+
3+
<!--
4+
This source file is part of the Swift.org open source project
5+
6+
Copyright (c) 2024 Apple Inc. and the Swift project authors
7+
Licensed under Apache License v2.0 with Runtime Library Exception
8+
9+
See https://swift.org/LICENSE.txt for license information
10+
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
11+
-->
12+
13+
This document describes the format and location of test content that the testing
14+
library emits at compile time and can discover at runtime.
15+
16+
## Basic format
17+
18+
Swift Testing uses the [ELF Note format](https://man7.org/linux/man-pages/man5/elf.5.html)
19+
to store individual records of test content. Records created and discoverable by
20+
the testing library are stored in dedicated platform-specific sections:
21+
22+
| Platform | Binary Format | Section Name |
23+
|-|:-:|-|
24+
| macOS | Mach-O | `__DATA_CONST,__swift5_tests` |
25+
| iOS | Mach-O | `__DATA_CONST,__swift5_tests` |
26+
| watchOS | Mach-O | `__DATA_CONST,__swift5_tests` |
27+
| tvOS | Mach-O | `__DATA_CONST,__swift5_tests` |
28+
| visionOS | Mach-O | `__DATA_CONST,__swift5_tests` |
29+
| Linux | ELF | `PT_NOTE`[^1] |
30+
| FreeBSD | ELF | `PT_NOTE`[^1] |
31+
| Android | ELF | `PT_NOTE`[^1] |
32+
| WASI | Statically Linked | `swift5_tests` |
33+
| Windows | PE/COFF | `.sw5test` |
34+
35+
[^1]: On platforms that use the ELF binary format natively, test content records
36+
are stored in ELF program headers of type `PT_NOTE`. Take care not to
37+
remove these program headers (for example, by invoking [`strip(1)`](https://www.man7.org/linux/man-pages/man1/strip.1.html).)
38+
39+
### Determining the type of test content
40+
41+
Regardless of platform, all test content records created and discoverable by the
42+
testing library start have the name `"Swift Testing"` stored in the implied
43+
`n_name` field of their underlying ELF Notes. Each record's _type_ (stored in
44+
the underlying ELF Note's `n_type` field) determines how the record will be
45+
interpreted at runtime:
46+
47+
| Type Value | Interpretation |
48+
|-:|-|
49+
| < `0` | Undefined (**do not use**) |
50+
| `0` ... `99` | Reserved |
51+
| `100` | Test or suite declaration |
52+
| `101` | Exit test |
53+
54+
<!-- When adding cases to this enumeration, be sure to also update the
55+
corresponding enumeration in Discovery.h and TestContentGeneration.swift. -->
56+
57+
### Loading test content from a record
58+
59+
For all currently-defined record types, the header and name are followed by a
60+
structure of the following form:
61+
62+
```c
63+
struct SWTTestContent {
64+
bool (* _Nonnull accessor)(void *);
65+
uint64_t flags;
66+
};
67+
```
68+
69+
#### The accessor field
70+
71+
The function `accessor` is a C function whose signature in Swift can be restated
72+
as:
73+
74+
```swift
75+
@convention(c) (_ outValue: UnsafeMutableRawPointer) -> Bool
76+
```
77+
78+
When called, it initializes the memory at `outValue` to an instance of some
79+
Swift type and returns `true`, or returns `false` if it could not generate the
80+
relevant content. On successful return, the caller is responsible for
81+
deinitializing the memory at `outValue` when done with it.
82+
83+
The concrete Swift type of `accessor`'s result depends on the type of record:
84+
85+
| Type Value | Return Type |
86+
|-:|-|
87+
| < `0` | Undefined (**do not use**) |
88+
| `0` ... `99` | `nil` |
89+
| `100` | `@Sendable () async -> Test`[^2] |
90+
| `101` | `ExitTest` (owned by caller) |
91+
92+
[^2]: This signature is not the signature of `accessor`, but of the Swift
93+
function reference it returns. This level of indirection is necessary
94+
because loading a test or suite declaration is an asynchronous operation,
95+
but C functions cannot be `async`.
96+
97+
#### The flags field
98+
99+
For test or suite declarations (type `100`), the following flags are defined:
100+
101+
| Bit | Description |
102+
|-:|-|
103+
| `1 << 0` | This record contains a suite declaration |
104+
| `1 << 1` | This record contains a parameterized test function declaration |
105+
106+
For exit test declarations (type `101`), no flags are currently defined.
107+
108+
## Third-party test content
109+
110+
TODO: elaborate how tools can reuse the same `n_name` and `n_type` fields to
111+
supplement Swift Testing's data, or use a different `n_name` field to store
112+
arbitrary other data in the test content section that Swift Testing will ignore.

Package.swift

+2
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
128128
.enableExperimentalFeature("AccessLevelOnImport"),
129129
.enableUpcomingFeature("InternalImportsByDefault"),
130130

131+
.enableExperimentalFeature("SymbolLinkageMarkers"),
132+
131133
.define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
132134

133135
.define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),

Sources/Testing/ExitTests/ExitTest.swift

+25-31
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,29 @@ public struct ExitTest: Sendable, ~Copyable {
2424
/// The expected exit condition of the exit test.
2525
public var expectedExitCondition: ExitCondition
2626

27-
/// The body closure of the exit test.
28-
fileprivate var body: @Sendable () async throws -> Void = {}
29-
3027
/// The source location of the exit test.
3128
///
3229
/// The source location is unique to each exit test and is consistent between
3330
/// processes, so it can be used to uniquely identify an exit test at runtime.
3431
public var sourceLocation: SourceLocation
3532

33+
/// The body closure of the exit test.
34+
fileprivate var body: @Sendable () async throws -> Void
35+
36+
/// Initialize an exit test at runtime.
37+
///
38+
/// - Warning: This initializer is used to implement the `#expect(exitsWith:)`
39+
/// macro. Do not use it directly.
40+
public init(
41+
__expectedExitCondition expectedExitCondition: ExitCondition,
42+
sourceLocation: SourceLocation,
43+
body: @escaping @Sendable () async throws -> Void = {}
44+
) {
45+
self.expectedExitCondition = expectedExitCondition
46+
self.sourceLocation = sourceLocation
47+
self.body = body
48+
}
49+
3650
/// Disable crash reporting, crash logging, or core dumps for the current
3751
/// process.
3852
private static func _disableCrashReporting() {
@@ -102,44 +116,24 @@ public struct ExitTest: Sendable, ~Copyable {
102116

103117
// MARK: - Discovery
104118

105-
/// A protocol describing a type that contains an exit test.
106-
///
107-
/// - Warning: This protocol is used to implement the `#expect(exitsWith:)`
108-
/// macro. Do not use it directly.
109-
@_alwaysEmitConformanceMetadata
110-
@_spi(Experimental)
111-
public protocol __ExitTestContainer {
112-
/// The expected exit condition of the exit test.
113-
static var __expectedExitCondition: ExitCondition { get }
114-
115-
/// The source location of the exit test.
116-
static var __sourceLocation: SourceLocation { get }
117-
118-
/// The body function of the exit test.
119-
static var __body: @Sendable () async throws -> Void { get }
120-
}
121-
122119
extension ExitTest {
123-
/// A string that appears within all auto-generated types conforming to the
124-
/// `__ExitTestContainer` protocol.
125-
private static let _exitTestContainerTypeNameMagic = "__🟠$exit_test_body__"
126-
127120
/// Find the exit test function at the given source location.
128121
///
129122
/// - Parameters:
130123
/// - sourceLocation: The source location of the exit test to find.
131124
///
132125
/// - Returns: The specified exit test function, or `nil` if no such exit test
133126
/// could be found.
127+
@_spi(ForToolsIntegrationOnly)
134128
public static func find(at sourceLocation: SourceLocation) -> Self? {
135129
var result: Self?
136130

137-
enumerateTypes(withNamesContaining: _exitTestContainerTypeNameMagic) { _, type, stop in
138-
if let type = type as? any __ExitTestContainer.Type, type.__sourceLocation == sourceLocation {
131+
enumerateTestContent(ofKind: .exitTest, as: ExitTest.self) { _, exitTest, _, stop in
132+
if exitTest.sourceLocation == sourceLocation {
139133
result = ExitTest(
140-
expectedExitCondition: type.__expectedExitCondition,
141-
body: type.__body,
142-
sourceLocation: type.__sourceLocation
134+
__expectedExitCondition: exitTest.expectedExitCondition,
135+
sourceLocation: exitTest.sourceLocation,
136+
body: exitTest.body
143137
)
144138
stop = true
145139
}
@@ -183,7 +177,7 @@ func callExitTest(
183177

184178
let actualExitCondition: ExitCondition
185179
do {
186-
let exitTest = ExitTest(expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
180+
let exitTest = ExitTest(__expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
187181
actualExitCondition = try await configuration.exitTestHandler(exitTest)
188182
} catch {
189183
// An error here would indicate a problem in the exit test handler such as a
@@ -295,7 +289,7 @@ extension ExitTest {
295289
// External tools authors should set up their own back channel mechanisms
296290
// and ensure they're installed before calling ExitTest.callAsFunction().
297291
guard var result = find(at: sourceLocation) else {
298-
return nil
292+
fatalError("Could not find an exit test that should have been located at \(sourceLocation).")
299293
}
300294

301295
// We can't say guard let here because it counts as a consume.

Sources/Testing/Test+Discovery.swift

+60-40
Original file line numberDiff line numberDiff line change
@@ -8,23 +8,9 @@
88
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
99
//
1010

11-
private import _TestingInternals
12-
13-
/// A protocol describing a type that contains tests.
14-
///
15-
/// - Warning: This protocol is used to implement the `@Test` macro. Do not use
16-
/// it directly.
17-
@_alwaysEmitConformanceMetadata
18-
public protocol __TestContainer {
19-
/// The set of tests contained by this type.
20-
static var __tests: [Test] { get async }
21-
}
11+
internal import _TestingInternals
2212

2313
extension Test {
24-
/// A string that appears within all auto-generated types conforming to the
25-
/// `__TestContainer` protocol.
26-
private static let _testContainerTypeNameMagic = "__🟠$test_container__"
27-
2814
/// All available ``Test`` instances in the process, according to the runtime.
2915
///
3016
/// The order of values in this sequence is unspecified.
@@ -47,16 +33,14 @@ extension Test {
4733
/// contain duplicates; callers should use ``all`` instead.
4834
private static var _all: some Sequence<Self> {
4935
get async {
50-
await withTaskGroup(of: [Self].self) { taskGroup in
51-
enumerateTypes(withNamesContaining: _testContainerTypeNameMagic) { _, type, _ in
52-
if let type = type as? any __TestContainer.Type {
53-
taskGroup.addTask {
54-
await type.__tests
55-
}
36+
await withTaskGroup(of: Self.self) { taskGroup in
37+
enumerateTestContent(ofKind: .testDeclaration, as: (@Sendable () async -> Test).self) { _, generator, _, _ in
38+
taskGroup.addTask {
39+
await generator()
5640
}
5741
}
5842

59-
return await taskGroup.reduce(into: [], +=)
43+
return await taskGroup.reduce(into: []) { $0.append($1) }
6044
}
6145
}
6246
}
@@ -111,35 +95,71 @@ extension Test {
11195

11296
// MARK: -
11397

114-
/// The type of callback called by ``enumerateTypes(withNamesContaining:_:)``.
98+
/// The type of callback called by `_enumerateTestContent(_:)`.
99+
///
100+
/// - Parameters:
101+
/// - record: A pointer to a structure containing information about the
102+
/// enumerated test content.
103+
/// - stop: A pointer to a boolean variable indicating whether test content
104+
/// enumeration should stop after the function returns. Set `*stop` to
105+
/// `true` to stop test content enumeration.
106+
private typealias _TestContentEnumerator = (_ record: UnsafePointer<SWTTestContentRecord>, _ stop: UnsafeMutablePointer<Bool>) -> Void
107+
108+
/// Enumerate all test content known to Swift and found in the current process.
109+
///
110+
/// - Parameters:
111+
/// - body: A function to invoke, once per raw test content record.
112+
///
113+
/// This function enumerates all raw test content records discovered at runtime.
114+
/// Callers should prefer ``enumerateTestContent(ofKind:as:_:)`` instead.
115+
private func _enumerateTestContent(_ body: _TestContentEnumerator) {
116+
withoutActuallyEscaping(body) { body in
117+
withUnsafePointer(to: body) { context in
118+
swt_enumerateTestContent(.init(mutating: context)) { record, stop, context in
119+
let body = context!.load(as: _TestContentEnumerator.self)
120+
body(record, stop)
121+
}
122+
}
123+
}
124+
}
125+
126+
/// The type of callback called by ``enumerateTestContent(ofKind:as:_:)``.
115127
///
116128
/// - Parameters:
117129
/// - imageAddress: A pointer to the start of the image. This value is _not_
118130
/// equal to the value returned from `dlopen()`. On platforms that do not
119-
/// support dynamic loading (and so do not have loadable images), this
120-
/// argument is unspecified.
121-
/// - type: A Swift type.
131+
/// support dynamic loading (and so do not have loadable images), the value
132+
/// of this argument is unspecified.
133+
/// - content: The enumerated test content.
134+
/// - flags: Flags associated with `content`. The value of this argument is
135+
/// dependent on the type of test content being enumerated.
122136
/// - stop: An `inout` boolean variable indicating whether type enumeration
123137
/// should stop after the function returns. Set `stop` to `true` to stop
124138
/// type enumeration.
125-
typealias TypeEnumerator = (_ imageAddress: UnsafeRawPointer?, _ type: Any.Type, _ stop: inout Bool) -> Void
139+
typealias TestContentEnumerator<T> = (_ imageAddress: UnsafeRawPointer?, _ content: borrowing T, _ flags: UInt64, _ stop: inout Bool) -> Void where T: ~Copyable
126140

127-
/// Enumerate all types known to Swift found in the current process whose names
128-
/// contain a given substring.
141+
/// Enumerate all test content known to Swift and found in the current process.
129142
///
130143
/// - Parameters:
131-
/// - nameSubstring: A string which the names of matching classes all contain.
132-
/// - body: A function to invoke, once per matching type.
133-
func enumerateTypes(withNamesContaining nameSubstring: String, _ typeEnumerator: TypeEnumerator) {
134-
withoutActuallyEscaping(typeEnumerator) { typeEnumerator in
135-
withUnsafePointer(to: typeEnumerator) { context in
136-
swt_enumerateTypes(withNamesContaining: nameSubstring, .init(mutating: context)) { imageAddress, type, stop, context in
137-
let typeEnumerator = context!.load(as: TypeEnumerator.self)
138-
let type = unsafeBitCast(type, to: Any.Type.self)
139-
var stop2 = false
140-
typeEnumerator(imageAddress, type, &stop2)
141-
stop.pointee = stop2
144+
/// - kind: The kind of test content to look for.
145+
/// - type: The Swift type of test content to look for.
146+
/// - body: A function to invoke, once per matching test content record.
147+
func enumerateTestContent<T>(ofKind kind: SWTTestContentKind, as type: T.Type, _ body: TestContentEnumerator<T>) where T: ~Copyable {
148+
_enumerateTestContent { record, stop in
149+
if record.pointee.kind != kind {
150+
return
151+
}
152+
withUnsafeTemporaryAllocation(of: type, capacity: 1) { buffer in
153+
// Load the content from the record via its accessor function.
154+
guard record.pointee.accessor(buffer.baseAddress!) else {
155+
return
142156
}
157+
defer {
158+
buffer.deinitialize()
159+
}
160+
161+
// Call the callback.
162+
body(record.pointee.imageAddress, buffer.baseAddress!.pointee, record.pointee.flags, &stop.pointee)
143163
}
144164
}
145165
}

Sources/Testing/Test+Macro.swift

+4-4
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@
88
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
99
//
1010

11+
/// This file provides support for the `@Test` macro. Other than the macro
12+
/// itself, the symbols in this file should not be used directly and are subject
13+
/// to change as the testing library evolves.
14+
1115
#if _runtime(_ObjC)
1216
public import ObjectiveC
1317

@@ -42,10 +46,6 @@ public typealias __XCTestCompatibleSelector = Never
4246
#endif
4347
}
4448

45-
/// This file provides support for the `@Test` macro. Other than the macro
46-
/// itself, the symbols in this file should not be used directly and are subject
47-
/// to change as the testing library evolves.
48-
4949
// MARK: - @Suite
5050

5151
/// Declare a test suite.

Sources/TestingMacros/CMakeLists.txt

+1
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ target_sources(TestingMacros PRIVATE
102102
Support/DiagnosticMessage+Diagnosing.swift
103103
Support/SourceCodeCapturing.swift
104104
Support/SourceLocationGeneration.swift
105+
Support/TestContentGeneration.swift
105106
TagMacro.swift
106107
TestDeclarationMacro.swift
107108
TestingMacrosMain.swift)

0 commit comments

Comments
 (0)