Skip to content

Commit a5eeaa7

Browse files
grynspangrynspan
committed
Store test content in a custom metadata section.
See also: swiftlang/swift#76698 Resolves #735.
1 parent 7dd3d27 commit a5eeaa7

19 files changed

+791
-447
lines changed

‎Documentation/ABI/TestContent.md

+175
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Runtime-discoverable test content
2+
3+
<!--
4+
This source file is part of the Swift.org open source project
5+
6+
Copyright (c) 2024 Apple Inc. and the Swift project authors
7+
Licensed under Apache License v2.0 with Runtime Library Exception
8+
9+
See https://swift.org/LICENSE.txt for license information
10+
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
11+
-->
12+
13+
This document describes the format and location of test content that the testing
14+
library emits at compile time and can discover at runtime.
15+
16+
## Basic format
17+
18+
Swift Testing uses the [ELF Note format](https://man7.org/linux/man-pages/man5/elf.5.html)
19+
to store individual records of test content. Records created and discoverable by
20+
the testing library are stored in dedicated platform-specific sections:
21+
22+
| Platform | Binary Format | Section Name |
23+
|-|:-:|-|
24+
| macOS, iOS, watchOS, tvOS, visionOS | Mach-O | `__DATA_CONST,__swift5_tests` |
25+
| Linux, FreeBSD, Android | ELF | `PT_NOTE`[^1] |
26+
| WASI | Statically Linked | `swift5_tests` |
27+
| Windows | PE/COFF | `.sw5test` |
28+
29+
[^1]: On platforms that use the ELF binary format natively, test content records
30+
are stored in ELF program headers of type `PT_NOTE`. Take care not to
31+
remove these program headers (for example, by invoking [`strip(1)`](https://www.man7.org/linux/man-pages/man1/strip.1.html).)
32+
33+
### Record headers
34+
35+
Regardless of platform, all test content records created and discoverable by the
36+
testing library have the following structure:
37+
38+
```c
39+
struct SWTTestContentHeader {
40+
int32_t n_namesz;
41+
int32_t n_descsz;
42+
int32_t n_type;
43+
char n_name[n_namesz];
44+
// ...
45+
};
46+
```
47+
48+
This structure can be represented in Swift as a heterogenous tuple:
49+
50+
```swift
51+
typealias SWTTestContentHeader = (
52+
n_namesz: Int32,
53+
n_descsz: Int32,
54+
n_type: Int32,
55+
n_name: (CChar, CChar, /* ... */),
56+
// ...
57+
)
58+
```
59+
60+
The size of `n_name` is dynamic and cannot be statically computed. The testing
61+
library always generates the name `"Swift Testing"` and specifies an `n_namesz`
62+
value of `20` (the string being null-padded to the correct length), but other
63+
content may be present in the same section whose header size differs. For more
64+
information about this structure such as its alignment requirements, see the
65+
documentation for the [ELF format](https://man7.org/linux/man-pages/man5/elf.5.html).
66+
67+
Each record's _kind_ (stored in the `n_type` field) determines how the record
68+
will be interpreted at runtime:
69+
70+
| Type Value | Interpretation |
71+
|-:|-|
72+
| `< 0` | Undefined (**do not use**) |
73+
| `0 ... 99` | Reserved |
74+
| `100` | Test or suite declaration |
75+
| `101` | Exit test |
76+
77+
<!-- When adding cases to this enumeration, be sure to also update the
78+
corresponding enumeration in Discovery.h and TestContentGeneration.swift. -->
79+
80+
### Record contents
81+
82+
For all currently-defined record types, the header structure is immediately
83+
followed by the actual content of the record. A test content record currently
84+
contains an `accessor` function to load the corresponding Swift content and a
85+
`flags` field whose value depends on the type of record. The overall structure
86+
of a record therefore looks like:
87+
88+
```c
89+
struct SWTTestContent {
90+
SWTTestContentHeader header;
91+
bool (* accessor)(void *outValue);
92+
uint32_t flags;
93+
uint32_t reserved;
94+
};
95+
```
96+
97+
Or, in Swift as a tuple:
98+
99+
```swift
100+
typealias SWTTestContent = (
101+
header: SWTTestContentHeader,
102+
accessor: @convention(c) (_ outValue: UnsafeMutableRawPointer) -> Bool,
103+
flags: UInt32,
104+
reserved: UInt32
105+
)
106+
```
107+
108+
This structure may grow in the future as needed. Check the `header.n_descsz`
109+
field to determine if there are additional fields present. Do not assume that
110+
the size of this structure will remain fixed over time or that all discovered
111+
test content records are the same size.
112+
113+
#### The accessor field
114+
115+
The function `accessor` is a C function. When called, it initializes the memory
116+
at its argument `outValue` to an instance of some Swift type and returns `true`,
117+
or returns `false` if it could not generate the relevant content. On successful
118+
return, the caller is responsible for deinitializing the memory at `outValue`
119+
when done with it.
120+
121+
The concrete Swift type of the value written to `outValue` depends on the type
122+
of record:
123+
124+
| Type Value | Return Type |
125+
|-:|-|
126+
| `..< 0` | Undefined (**do not use**) |
127+
| `0 ... 99` | Reserved (**do not use**) |
128+
| `100` | `@Sendable () async -> Test`[^2] |
129+
| `101` | `ExitTest` (consumed by caller) |
130+
131+
[^2]: This signature is not the signature of `accessor`, but of the Swift
132+
function reference it writes to `outValue`. This level of indirection is
133+
necessary because loading a test or suite declaration is an asynchronous
134+
operation, but C functions cannot be `async`.
135+
136+
#### The flags field
137+
138+
For test or suite declarations (type `100`), the following flags are defined:
139+
140+
| Bit | Description |
141+
|-:|-|
142+
| `1 << 0` | This record contains a suite declaration |
143+
| `1 << 1` | This record contains a parameterized test function declaration |
144+
145+
For exit test declarations (type `101`), no flags are currently defined.
146+
147+
#### The reserved field
148+
149+
This field is reserved for future use. Always set it to `0`.
150+
151+
## Third-party test content
152+
153+
Testing tools may make use of the same storage and discovery mechanisms by
154+
emitting their own test content records into the test record content section.
155+
156+
Third-party test content should use the same value for the `n_name` field
157+
(`"Swift Testing"`). The `n_type` field should be set to a unique value only
158+
used by that tool, or used by that tool in collaboration with other compatible
159+
tools. At runtime, Swift Testing ignores test content records with unrecognized
160+
`n_type` values. To reserve a new unique `n_type` value, open a [GitHub issue](https://github.com/swiftlang/swift-testing/issues/new/choose)
161+
against Swift Testing.
162+
163+
The layout of third-party test content records must be compatible with that of
164+
`SWTTestContentHeader` as specified above. For the actual content of a test
165+
record, you do not need to use the same on-disk/in-memory layout as is specified
166+
by `SWTTestContent` above, but it is preferred. Third-party tools are ultimately
167+
responsible for ensuring the values they emit into the test content section are
168+
correctly aligned and have sufficient padding; failure to do so may render
169+
downstream test code unusable.
170+
171+
<!--
172+
TODO: elaborate further, give examples
173+
TODO: standardize a mechanism for third parties to produce `Test` instances
174+
since we don't have a public initializer for the `Test` type.
175+
-->

‎Documentation/Porting.md

+17-11
Original file line numberDiff line numberDiff line change
@@ -113,13 +113,14 @@ Once the header is included, we can call `GetDateTime()` from `Clock.swift`:
113113
## Runtime test discovery
114114

115115
When porting to a new platform, you may need to provide a new implementation for
116-
`enumerateTypeMetadataSections()` in `Discovery.cpp`. Test discovery is
117-
dependent on Swift metadata discovery which is an inherently platform-specific
118-
operation.
116+
`enumerateTestContentSections()` in `Discovery.cpp`. Test discovery is dependent
117+
on Swift metadata discovery which is an inherently platform-specific operation.
119118

120-
_Most_ platforms will be able to reuse the implementation used by Linux and
121-
Windows that calls an internal Swift runtime function to enumerate available
122-
metadata. If you are porting Swift Testing to Classic, this function won't be
119+
_Most_ platforms in use today use the ELF image format and will be able to reuse
120+
the implementation used by Linux. That implementation calls `dl_iterate_phdr()`
121+
in the GNU C Library to enumerate available metadata.
122+
123+
If you are porting Swift Testing to Classic, `dl_iterate_phdr()` won't be
123124
available, so you'll need to write a custom implementation instead. Assuming
124125
that the Swift compiler emits section information into the resource fork on
125126
Classic, you could use the [Resource Manager](https://developer.apple.com/library/archive/documentation/mac/pdf/MoreMacintoshToolbox.pdf)
@@ -132,16 +133,21 @@ to load that information:
132133
// ...
133134
+#elif defined(macintosh)
134135
+template <typename SectionEnumerator>
135-
+static void enumerateTypeMetadataSections(const SectionEnumerator& body) {
136+
+static void enumerateTestContentSections(const SectionEnumerator& body) {
136137
+ ResFileRefNum refNum;
137138
+ if (noErr == GetTopResourceFile(&refNum)) {
138139
+ ResFileRefNum oldRefNum = refNum;
139140
+ do {
140141
+ UseResFile(refNum);
141-
+ Handle handle = Get1NamedResource('swft', "\p__swift5_types");
142+
+ Handle handle = Get1NamedResource('swft', "\p__swift5_tests");
142143
+ if (handle && *handle) {
143-
+ size_t size = GetHandleSize(handle);
144-
+ body(*handle, size);
144+
+ auto imageAddress = reinterpret_cast<const void *>(static_cast<uintptr_t>(refNum));
145+
+ SWTSectionBounds sb = { imageAddress, *handle, GetHandleSize(handle) };
146+
+ bool stop = false;
147+
+ body(sb, &stop);
148+
+ if (stop) {
149+
+ break;
150+
+ }
145151
+ }
146152
+ } while (noErr == GetNextResourceFile(refNum, &refNum));
147153
+ UseResFile(oldRefNum);
@@ -150,7 +156,7 @@ to load that information:
150156
#else
151157
#warning Platform-specific implementation missing: Runtime test discovery unavailable
152158
template <typename SectionEnumerator>
153-
static void enumerateTypeMetadataSections(const SectionEnumerator& body) {}
159+
static void enumerateTestContentSections(const SectionEnumerator& body) {}
154160
#endif
155161
```
156162

‎Package.swift

+11
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
128128
.enableExperimentalFeature("AccessLevelOnImport"),
129129
.enableUpcomingFeature("InternalImportsByDefault"),
130130

131+
.enableExperimentalFeature("SymbolLinkageMarkers"),
132+
131133
.define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
132134

133135
.define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),
@@ -162,6 +164,15 @@ extension Array where Element == PackageDescription.CXXSetting {
162164
static var packageSettings: Self {
163165
var result = Self()
164166

167+
result += [
168+
.define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
169+
170+
.define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),
171+
.define("SWT_NO_SNAPSHOT_TYPES", .when(platforms: [.linux, .windows, .wasi])),
172+
.define("SWT_NO_DYNAMIC_LINKING", .when(platforms: [.wasi])),
173+
.define("SWT_NO_PIPES", .when(platforms: [.wasi])),
174+
]
175+
165176
// Capture the testing library's version as a C++ string constant.
166177
if let git = Context.gitInformation {
167178
let testingLibraryVersion = if let tag = git.currentTag {

‎Sources/Testing/ExitTests/ExitTest.swift

+39-36
Original file line numberDiff line numberDiff line change
@@ -17,22 +17,44 @@ private import _TestingInternals
1717

1818
/// A type describing an exit test.
1919
///
20-
/// Instances of this type describe an exit test defined by the test author and
21-
/// discovered or called at runtime.
22-
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
20+
/// An instance of this type describes an exit test defined in a test target and
21+
/// discovered or called at runtime. You do not create instances of this type.
22+
///
23+
/// You don't usually need to interact with an instance of this type. To create
24+
/// an exit test, use the ``expect(exitsWith:_:sourceLocation:performing:)``
25+
/// or ``require(exitsWith:_:sourceLocation:performing:)`` macro.
26+
@_spi(Experimental)
2327
public struct ExitTest: Sendable, ~Copyable {
24-
/// The expected exit condition of the exit test.
28+
/// This exit test's expected exit condition.
2529
public var expectedExitCondition: ExitCondition
2630

27-
/// The body closure of the exit test.
28-
fileprivate var body: @Sendable () async throws -> Void = {}
29-
30-
/// The source location of the exit test.
31+
/// The source location of this exit test.
3132
///
3233
/// The source location is unique to each exit test and is consistent between
3334
/// processes, so it can be used to uniquely identify an exit test at runtime.
3435
public var sourceLocation: SourceLocation
3536

37+
/// The body closure of the exit test.
38+
///
39+
/// Do not invoke this closure directly. Instead, invoke ``callAsFunction()``
40+
/// to run the exit test. Running the exit test will always terminate the
41+
/// current process.
42+
fileprivate var body: @Sendable () async throws -> Void
43+
44+
/// Initialize an exit test at runtime.
45+
///
46+
/// - Warning: This initializer is used to implement the `#expect(exitsWith:)`
47+
/// macro. Do not use it directly.
48+
public init(
49+
__expectedExitCondition expectedExitCondition: ExitCondition,
50+
sourceLocation: SourceLocation,
51+
body: @escaping @Sendable () async throws -> Void = {}
52+
) {
53+
self.expectedExitCondition = expectedExitCondition
54+
self.sourceLocation = sourceLocation
55+
self.body = body
56+
}
57+
3658
/// Disable crash reporting, crash logging, or core dumps for the current
3759
/// process.
3860
private static func _disableCrashReporting() {
@@ -83,6 +105,7 @@ public struct ExitTest: Sendable, ~Copyable {
83105
/// to terminate the process; if it does not, the testing library will
84106
/// terminate the process in a way that causes the corresponding expectation
85107
/// to fail.
108+
@_spi(ForToolsIntegrationOnly)
86109
public consuming func callAsFunction() async -> Never {
87110
Self._disableCrashReporting()
88111

@@ -102,44 +125,24 @@ public struct ExitTest: Sendable, ~Copyable {
102125

103126
// MARK: - Discovery
104127

105-
/// A protocol describing a type that contains an exit test.
106-
///
107-
/// - Warning: This protocol is used to implement the `#expect(exitsWith:)`
108-
/// macro. Do not use it directly.
109-
@_alwaysEmitConformanceMetadata
110-
@_spi(Experimental)
111-
public protocol __ExitTestContainer {
112-
/// The expected exit condition of the exit test.
113-
static var __expectedExitCondition: ExitCondition { get }
114-
115-
/// The source location of the exit test.
116-
static var __sourceLocation: SourceLocation { get }
117-
118-
/// The body function of the exit test.
119-
static var __body: @Sendable () async throws -> Void { get }
120-
}
121-
122128
extension ExitTest {
123-
/// A string that appears within all auto-generated types conforming to the
124-
/// `__ExitTestContainer` protocol.
125-
private static let _exitTestContainerTypeNameMagic = "__🟠$exit_test_body__"
126-
127129
/// Find the exit test function at the given source location.
128130
///
129131
/// - Parameters:
130132
/// - sourceLocation: The source location of the exit test to find.
131133
///
132134
/// - Returns: The specified exit test function, or `nil` if no such exit test
133135
/// could be found.
136+
@_spi(ForToolsIntegrationOnly)
134137
public static func find(at sourceLocation: SourceLocation) -> Self? {
135138
var result: Self?
136139

137-
enumerateTypes(withNamesContaining: _exitTestContainerTypeNameMagic) { _, type, stop in
138-
if let type = type as? any __ExitTestContainer.Type, type.__sourceLocation == sourceLocation {
140+
enumerateTestContent(ofKind: .exitTest, as: ExitTest.self) { _, exitTest, _, stop in
141+
if exitTest.sourceLocation == sourceLocation {
139142
result = ExitTest(
140-
expectedExitCondition: type.__expectedExitCondition,
141-
body: type.__body,
142-
sourceLocation: type.__sourceLocation
143+
__expectedExitCondition: exitTest.expectedExitCondition,
144+
sourceLocation: exitTest.sourceLocation,
145+
body: exitTest.body
143146
)
144147
stop = true
145148
}
@@ -183,7 +186,7 @@ func callExitTest(
183186

184187
let actualExitCondition: ExitCondition
185188
do {
186-
let exitTest = ExitTest(expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
189+
let exitTest = ExitTest(__expectedExitCondition: expectedExitCondition, sourceLocation: sourceLocation)
187190
actualExitCondition = try await configuration.exitTestHandler(exitTest)
188191
} catch {
189192
// An error here would indicate a problem in the exit test handler such as a
@@ -295,7 +298,7 @@ extension ExitTest {
295298
// External tools authors should set up their own back channel mechanisms
296299
// and ensure they're installed before calling ExitTest.callAsFunction().
297300
guard var result = find(at: sourceLocation) else {
298-
return nil
301+
fatalError("Could not find an exit test that should have been located at \(sourceLocation).")
299302
}
300303

301304
// We can't say guard let here because it counts as a consume.

0 commit comments

Comments
 (0)