Skip to content

Commit bc3e04b

Browse files
grynspangrynspan
committed
Store test content in a custom metadata section.
This PR uses the experimental symbol linkage margers feature in the Swift compiler to emit metadata about tests (and exit tests) into a dedicated section of the test executable being built. At runtime, we discover that section and read out the tests from it. This has several benefits over our current model, which involves walking Swift's type metadata table looking for types that conform to a protocol: 1. We don't need to define that protocol as public API in Swift Testing, 1. We don't need to emit type metadata (much larger than what we really need) for every test function, 1. We don't need to duplicate a large chunk of the Swift ABI sources in order to walk the type metadata table correctly, and 1. Almost all the new code is written in Swift, whereas the code it is intended to replace could not be fully represented in Swift and needed to be written in C++. The change also opens up the possibility of supporting generic types in the future because we can emit metadata without needing to emit a nested type (which is not always valid in a generic context.) That's a "future direction" and not covered by this PR specifically. I've defined a layout for entries in the new `swift5_tests` section that should be flexible enough for us in the short-to-medium term and which lets us define additional arbitrary test content record types. The layout of this section is covered in depth in the new [TestContent.md](Documentation/ABI/TestContent.md) article. This functionality is only available if a test target enables the experimental `"SymbolLinkageMarkers"` feature. We continue to emit protocol-conforming types for now—that code will be removed if and when the experimental feature is properly supported (modulo us adopting relevant changes to the feature's API.) #735 swiftlang/swift#76698 swiftlang/swift#78411
1 parent 878df30 commit bc3e04b

32 files changed

+1489
-195
lines changed

‎Documentation/ABI/TestContent.md

+178
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
# Runtime-discoverable test content
2+
3+
<!--
4+
This source file is part of the Swift.org open source project
5+
6+
Copyright (c) 2024 Apple Inc. and the Swift project authors
7+
Licensed under Apache License v2.0 with Runtime Library Exception
8+
9+
See https://swift.org/LICENSE.txt for license information
10+
See https://swift.org/CONTRIBUTORS.txt for Swift project authors
11+
-->
12+
13+
This document describes the format and location of test content that the testing
14+
library emits at compile time and can discover at runtime.
15+
16+
> [!WARNING]
17+
> The content of this document is subject to change pending efforts to define a
18+
> Swift-wide standard mechanism for runtime metadata emission and discovery.
19+
> Treat the information in this document as experimental.
20+
21+
## Basic format
22+
23+
Swift Testing stores test content records in a dedicated platform-specific
24+
section in built test products:
25+
26+
| Platform | Binary Format | Section Name |
27+
|-|:-:|-|
28+
| macOS, iOS, watchOS, tvOS, visionOS | Mach-O | `__DATA_CONST,__swift5_tests` |
29+
| Linux, FreeBSD, Android | ELF | `swift5_tests` |
30+
| WASI | Statically Linked | `swift5_tests`[^1] |
31+
| Windows | PE/COFF | `.sw5test`[^2] |
32+
33+
[^1]: SwiftWasm effectively uses the ELF format for its images, however it is
34+
currently always statically linked and runtime discovery is performed
35+
using a different mechanism than what we use on other ELF-based platforms.
36+
[^2]: On Windows, the Swift compiler [emits](https://github.com/swiftlang/swift/blob/main/stdlib/public/runtime/SwiftRT-COFF.cpp)
37+
leading and trailing padding into this section, both zeroed and of size
38+
`MemoryLayout<UInt>.stride`. Code that walks this section can safely skip
39+
over this padding.
40+
41+
### Record headers
42+
43+
Regardless of platform, all test content records created and discoverable by the
44+
testing library have the following layout:
45+
46+
```swift
47+
typealias TestContentRecord = (
48+
kind: UInt32,
49+
version: UInt16,
50+
reserved1: UInt16,
51+
accessor: (@convention(c) (_ outValue: UnsafeMutableRawPointer, _ hint: UnsafeRawPointer?) -> CBool)?,
52+
context: UInt,
53+
reserved2: UInt
54+
)
55+
```
56+
57+
This type has natural size, stride, and alignment. Its fields are native-endian.
58+
If needed, this type can be represented in C as a structure:
59+
60+
```c
61+
struct SWTTestContentRecord {
62+
uint32_t kind;
63+
uint16_t version;
64+
uint16_t reserved1;
65+
bool (* _Nullable accessor)(void *outValue, const void *_Null_unspecified hint);
66+
uintptr_t context;
67+
uintptr_t reserved2;
68+
};
69+
```
70+
71+
### Record contents
72+
73+
#### The kind field
74+
75+
Each record's _kind_ determines how the record will be interpreted at runtime. A
76+
record's kind is a 32-bit unsigned value. The following kinds are defined:
77+
78+
| As Hexadecimal | As [FourCC](https://en.wikipedia.org/wiki/FourCC) | Interpretation |
79+
|-:|:-:|-|
80+
| `0x00000000` | &ndash; | Reserved (**do not use**) |
81+
| `0x74657374` | `'test'` | Test or suite declaration |
82+
| `0x65786974` | `'exit'` | Exit test |
83+
84+
<!-- When adding cases to this enumeration, be sure to also update the
85+
corresponding enumeration in TestContentGeneration.swift. -->
86+
87+
#### The version field
88+
89+
This field is currently always `0`. Implementations should ignore structures
90+
with other version values.
91+
92+
#### The accessor field
93+
94+
The function `accessor` is a C function. When called, it initializes the memory
95+
at its argument `outValue` to an instance of some Swift type and returns `true`,
96+
or returns `false` if it could not generate the relevant content. On successful
97+
return, the caller is responsible for deinitializing the memory at `outValue`
98+
when done with it.
99+
100+
`accessor` is optional. If it is `nil`, the test content record is ignored. The
101+
testing library may, in the future, define record kinds that do not provide an
102+
accessor function (that is, they represent pure compile-time information only.)
103+
104+
The second argument to this function, `hint`, is an optional input that can be
105+
passed to help the accessor function determine if its corresponding test content
106+
record matches what the caller is looking for. If the caller passes `nil` as the
107+
`hint` argument, the accessor behaves as if it matched (that is, no additional
108+
filtering is performed.)
109+
110+
The concrete Swift type of the value written to `outValue` and the value pointed
111+
to by `hint` depend on the kind of record:
112+
113+
- For test or suite declarations (kind `0x74657374`), the accessor produces an
114+
asynchronous Swift function that returns an instance of `Test`:
115+
116+
```swift
117+
@Sendable () async -> Test
118+
```
119+
120+
This signature is not the signature of `accessor`, but of the Swift function
121+
reference it writes to `outValue`. This level of indirection is necessary
122+
because loading a test or suite declaration is an asynchronous operation, but
123+
C functions cannot be `async`.
124+
125+
Test content records of this kind do not specify a type for `hint`. Always
126+
pass `nil`.
127+
128+
- For exit test declarations (kind `0x65786974`), the accessor produces a
129+
structure describing the exit test (of type `__ExitTest`.)
130+
131+
Test content records of this kind accept a `hint` of type `SourceLocation`.
132+
They only produce a result if they represent an exit test declared at the same
133+
source location (or if the hint is `nil`.)
134+
135+
#### The context field
136+
137+
This field can be used by test content to store additional context for a test
138+
content record that needs to be made available before the accessor is called:
139+
140+
- For test or suite declarations (kind `0x74657374`), this field contains a bit
141+
mask with the following flags currently defined:
142+
143+
| Bit | Description |
144+
|-:|-|
145+
| `1 << 0` | This record contains a suite declaration |
146+
| `1 << 1` | This record contains a parameterized test function declaration |
147+
148+
Other bits are currently always set to `0`, but may be used in the future.
149+
150+
- For exit test declarations (kind `0x65786974`), this field is not used and
151+
should be set to `0`.
152+
153+
#### The reserved1 and reserved2 fields
154+
155+
These fields are reserved for future use. Always set them to `0`.
156+
157+
## Third-party test content
158+
159+
Testing tools may make use of the same storage and discovery mechanisms by
160+
emitting their own test content records into the test record content section.
161+
162+
Third-party test content should set the `kind` field to a unique value only used
163+
by that tool, or used by that tool in collaboration with other compatible tools.
164+
At runtime, Swift Testing ignores test content records with unrecognized `kind`
165+
values. To reserve a new unique `kind` value, open a [GitHub issue](https://github.com/swiftlang/swift-testing/issues/new/choose)
166+
against Swift Testing.
167+
168+
The layout of third-party test content records must be compatible with that of
169+
`TestContentRecord` as specified above. Third-party tools are ultimately
170+
responsible for ensuring the values they emit into the test content section are
171+
correctly aligned and have sufficient padding; failure to do so may render
172+
downstream test code unusable.
173+
174+
<!--
175+
TODO: elaborate further, give examples
176+
TODO: standardize a mechanism for third parties to produce `Test` instances
177+
since we don't have a public initializer for the `Test` type.
178+
-->

‎Documentation/Porting.md

+78-41
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ platform-specific attention.
6666
> These errors are produced when the configuration you're trying to build has
6767
> conflicting requirements (for example, attempting to enable support for pipes
6868
> without also enabling support for file I/O.) You should be able to resolve
69-
> these issues by updating Package.swift and/or CompilerSettings.cmake.
69+
> these issues by updating `Package.swift` and/or `CompilerSettings.cmake`.
7070
7171
Most platform dependencies can be resolved through the use of platform-specific
7272
API. For example, Swift Testing uses the C11 standard [`timespec`](https://en.cppreference.com/w/c/chrono/timespec)
@@ -123,69 +123,106 @@ Once the header is included, we can call `GetDateTime()` from `Clock.swift`:
123123
## Runtime test discovery
124124

125125
When porting to a new platform, you may need to provide a new implementation for
126-
`enumerateTypeMetadataSections()` in `Discovery.cpp`. Test discovery is
126+
`_testContentSectionBounds()` in `Discovery+Platform.swift`. Test discovery is
127127
dependent on Swift metadata discovery which is an inherently platform-specific
128128
operation.
129129

130-
_Most_ platforms will be able to reuse the implementation used by Linux and
131-
Windows that calls an internal Swift runtime function to enumerate available
132-
metadata. If you are porting Swift Testing to Classic, this function won't be
130+
> [!NOTE]
131+
> You do not need to provide an implementation for the function
132+
> `enumerateTypeMetadataSections()` in `Discovery+Old.cpp`: it is present for
133+
> backwards compatibility with Swift 6.0 toolchains and will be removed in a
134+
> future release.
135+
136+
_Most_ platforms in use today use the ELF image format and will be able to reuse
137+
the implementation used by Linux. That implementation calls `dl_iterate_phdr()`
138+
in the GNU C Library to enumerate available metadata.
139+
140+
If you are porting Swift Testing to Classic, `dl_iterate_phdr()` won't be
133141
available, so you'll need to write a custom implementation instead. Assuming
134142
that the Swift compiler emits section information into the resource fork on
135-
Classic, you could use the [Resource Manager](https://developer.apple.com/library/archive/documentation/mac/pdf/MoreMacintoshToolbox.pdf)
143+
Classic, you would use the [Resource Manager](https://developer.apple.com/library/archive/documentation/mac/pdf/MoreMacintoshToolbox.pdf)
136144
to load that information:
137145

138146
```diff
139-
--- a/Sources/_TestingInternals/Discovery.cpp
140-
+++ b/Sources/_TestingInternals/Discovery.cpp
147+
--- a/Sources/Testing/Discovery+Platform.swift
148+
+++ b/Sources/Testing/Discovery+Platform.swift
141149

142150
// ...
143-
+#elif defined(macintosh)
144-
+template <typename SectionEnumerator>
145-
+static void enumerateTypeMetadataSections(const SectionEnumerator& body) {
146-
+ ResFileRefNum refNum;
147-
+ if (noErr == GetTopResourceFile(&refNum)) {
148-
+ ResFileRefNum oldRefNum = refNum;
149-
+ do {
150-
+ UseResFile(refNum);
151-
+ Handle handle = Get1NamedResource('swft', "\p__swift5_types");
152-
+ if (handle && *handle) {
153-
+ auto imageAddress = reinterpret_cast<const void *>(static_cast<uintptr_t>(refNum));
154-
+ SWTSectionBounds sb = { imageAddress, *handle, GetHandleSize(handle) };
155-
+ bool stop = false;
156-
+ body(sb, &stop);
157-
+ if (stop) {
158-
+ break;
159-
+ }
160-
+ }
161-
+ } while (noErr == GetNextResourceFile(refNum, &refNum));
162-
+ UseResFile(oldRefNum);
151+
+#elseif os(macintosh)
152+
+private func _testContentSectionBounds() -> [SectionBounds] {
153+
+ let oldRefNum = CurResFile()
154+
+ defer {
155+
+ UseResFile(oldRefNum)
156+
+ }
157+
+
158+
+ var refNum = ResFileRefNum(0)
159+
+ guard noErr == GetTopResourceFile(&refNum) else {
160+
+ return []
163161
+ }
162+
+
163+
+ var result = [SectionBounds]()
164+
+ repeat {
165+
+ UseResFile(refNum)
166+
+ guard let handle = Get1NamedResource(ResType("swft"), Str255("__swift5_tests")) else {
167+
+ continue
168+
+ }
169+
+ let sb = SectionBounds(
170+
+ imageAddress: UnsafeRawPointer(bitPattern: UInt(refNum)),
171+
+ start: handle.pointee!,
172+
+ size: GetHandleSize(handle)
173+
+ )
174+
+ result.append(sb)
175+
+ } while noErr == GetNextResourceFile(refNum, &refNum))
176+
+ return result
164177
+}
165178
#else
166-
#warning Platform-specific implementation missing: Runtime test discovery unavailable (dynamic)
167-
template <typename SectionEnumerator>
168-
static void enumerateTypeMetadataSections(const SectionEnumerator& body) {}
179+
private func _testContentSectionBounds() -> [SectionBounds] {
180+
#warning("Platform-specific implementation missing: Runtime test discovery unavailable (dynamic)")
181+
return []
182+
}
169183
#endif
170184
```
171185

186+
You will also need to update the `makeTestContentRecordDecl()` function in the
187+
`TestingMacros` target to emit the correct `@_section` attribute for your
188+
platform. If your platform uses the ELF image format and supports the
189+
`dl_iterate_phdr()` function, add it to the existing `#elseif os(Linux) || ...`
190+
case. Otherwise, add a new case for your platform:
191+
192+
```diff
193+
--- a/Sources/TestingMacros/Support/TestContentGeneration.swift
194+
+++ b/Sources/TestingMacros/Support/TestContentGeneration.swift
195+
// ...
196+
+ #elseif os(Classic)
197+
+ @_section(".rsrc,swft,__swift5_tests")
198+
#else
199+
@__testing(warning: "Platform-specific implementation missing: test content section name unavailable")
200+
#endif
201+
```
202+
203+
Keep in mind that this code is emitted by the `@Test` and `@Suite` macros
204+
directly into test authors' test targets, so you will not be able to use
205+
compiler conditionals defined in the Swift Testing package (including those that
206+
start with `"SWT_"`).
207+
172208
## Runtime test discovery with static linkage
173209

174210
If your platform does not support dynamic linking and loading, you will need to
175211
use static linkage instead. Define the `"SWT_NO_DYNAMIC_LINKING"` compiler
176-
conditional for your platform in both Package.swift and CompilerSettings.cmake,
177-
then define the `sectionBegin` and `sectionEnd` symbols in Discovery.cpp:
212+
conditional for your platform in both `Package.swift` and
213+
`CompilerSettings.cmake`, then define the `testContentSectionBegin` and
214+
`testContentSectionEnd` symbols in `Discovery.cpp`:
178215

179216
```diff
180217
diff --git a/Sources/_TestingInternals/Discovery.cpp b/Sources/_TestingInternals/Discovery.cpp
181218
// ...
182219
+#elif defined(macintosh)
183-
+extern "C" const char sectionBegin __asm__("...");
184-
+extern "C" const char sectionEnd __asm__("...");
220+
+extern "C" const char testContentSectionBegin __asm__("...");
221+
+extern "C" const char testContentSectionEnd __asm__("...");
185222
#else
186223
#warning Platform-specific implementation missing: Runtime test discovery unavailable (static)
187-
static const char sectionBegin = 0;
188-
static const char& sectionEnd = sectionBegin;
224+
static const char testContentSectionBegin = 0;
225+
static const char& testContentSectionEnd = testContentSectionBegin;
189226
#endif
190227
```
191228

@@ -204,12 +241,12 @@ diff --git a/Sources/_TestingInternals/Discovery.cpp b/Sources/_TestingInternals
204241
+#elif defined(macintosh)
205242
+extern "C" const char __linker_defined_begin_symbol;
206243
+extern "C" const char __linker_defined_end_symbol;
207-
+static const auto& sectionBegin = __linker_defined_begin_symbol;
208-
+static const auto& sectionEnd = __linker_defined_end_symbol;
244+
+static const auto& testContentSectionBegin = __linker_defined_begin_symbol;
245+
+static const auto& testContentSectionEnd = __linker_defined_end_symbol;
209246
#else
210247
#warning Platform-specific implementation missing: Runtime test discovery unavailable (static)
211-
static const char sectionBegin = 0;
212-
static const char& sectionEnd = sectionBegin;
248+
static const char testContentSectionBegin = 0;
249+
static const char& testContentSectionEnd = testContentSectionBegin;
213250
#endif
214251
```
215252

‎Package.swift

+2
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
152152
.enableExperimentalFeature("AccessLevelOnImport"),
153153
.enableUpcomingFeature("InternalImportsByDefault"),
154154

155+
.enableExperimentalFeature("SymbolLinkageMarkers"),
156+
155157
.define("SWT_TARGET_OS_APPLE", .when(platforms: [.macOS, .iOS, .macCatalyst, .watchOS, .tvOS, .visionOS])),
156158

157159
.define("SWT_NO_EXIT_TESTS", .when(platforms: [.iOS, .watchOS, .tvOS, .visionOS, .wasi, .android])),

‎Sources/Testing/CMakeLists.txt

+3
Original file line numberDiff line numberDiff line change
@@ -81,10 +81,13 @@ add_library(Testing
8181
Support/Locked.swift
8282
Support/SystemError.swift
8383
Support/Versions.swift
84+
Discovery.swift
85+
Discovery+Platform.swift
8486
Test.ID.Selection.swift
8587
Test.ID.swift
8688
Test.swift
8789
Test+Discovery.swift
90+
Test+Discovery+Old.swift
8891
Test+Macro.swift
8992
Traits/Bug.swift
9093
Traits/Comment.swift

0 commit comments

Comments
 (0)