Skip to content

Commit dac24fb

Browse files
grynspangrynspan
committed
Propose a stabilized C-compatible ABI interface and JSON schema for event streaming.
1 parent 8edb08f commit dac24fb

File tree

9 files changed

+442
-87
lines changed

9 files changed

+442
-87
lines changed

‎Documentation/ABI/JSON.md

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,9 @@ See https://swift.org/CONTRIBUTORS.txt for Swift project authors
1111
-->
1212

1313
This document outlines the JSON schemas used by the testing library for its ABI
14-
entry point and for the `--experimental-event-stream-output` command-line
15-
argument. For more information about the ABI entry point, see the documentation
16-
for [ABI.EntryPoint_v0](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint_v0&type=code).
17-
18-
> [!WARNING]
19-
> This JSON schema is still being developed and is subject to any and all
20-
> changes including removal from the package.
14+
entry point and for the `--event-stream-output` command-line argument. For more
15+
information about the ABI entry point, see the documentation for
16+
[ABIv0.EntryPoint](https://github.com/search?q=repo%3Aapple%2Fswift-testing%EntryPoint&type=code).
2117

2218
## Modified Backus-Naur form
2319

@@ -103,8 +99,8 @@ encoded as a single [JSON Lines](https://jsonlines.org) value.
10399

104100
A stream consists of a sequence of values encoded as [JSON Lines](https://jsonlines.org).
105101
A single instance of `<output-stream>` is defined per test process and can be
106-
accessed by passing `--experimental-event-stream-output` to the test executable
107-
created by `swift build --build-tests`.
102+
accessed by passing `--event-stream-output` to the test executable created by
103+
`swift build --build-tests`.
108104

109105
```
110106
<output-stream> ::= <output-record>\n | <output-record>\n <output-stream>

‎Documentation/Proposals/NNNN-json-abi.md

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# A stable JSON-based ABI for tools integration
2+
3+
* Proposal: [SWT-NNNN](NNNN-json-abi.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Status: **Awaiting review**
6+
* Implementation: [apple/swift-testing#383](https://github.com/apple/swift-testing/pull/383),
7+
[apple/swift-testing#402](https://github.com/apple/swift-testing/pull/402)
8+
<!-- * Review: ([pitch](https://forums.swift.org/...)) -->
9+
10+
11+
## Introduction
12+
13+
One of the core components of Swift Testing is its ability to interoperate with
14+
Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced
15+
across all platforms supported by Swift, and can be added as a package
16+
dependency (or—eventually—linked from the Swift toolchain.)
17+
18+
## Motivation
19+
20+
Because Swift Testing may be used in various forms, and because integration with
21+
various tools is critical to its success, we need it to have a stable interface
22+
that can be used regardless of how it's been added to a package. There are a few
23+
patterns in particular we know we need to support:
24+
25+
- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing:
26+
the copy used by the IDE might be the same as the copy that tests use, in
27+
which case interoperation is trivial, but it may also be distinct if the tests
28+
use Swift Testing as a package dependency.
29+
30+
In the case of Xcode 16, Swift Testing is built as a framework much like
31+
XCTest and is automatically linked by test targets in an Xcode project or
32+
Swift package, but if the test target specifies a package dependency on Swift
33+
Testing, that dependency will take priority when the test code is compiled.
34+
35+
- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and
36+
perhaps, as with VS Code, cannot because it is not natively compiled): such an
37+
IDE needs a way to configure and invoke test code and then to read events back
38+
as they occur, but cannot touch the Swift symbols used by the tests.
39+
40+
In the case of VS Code, because it is implemented using TypeScript, it is not
41+
able to directly link to Swift Testing or other Swift libraries. In order for
42+
it to interpret events from a test run like "test passed" or "issue recorded",
43+
it needs to receive those events in a format it can understand.
44+
45+
Tools integration is important to the success of Swift Testing. The more tools
46+
provide integrations for it, the more likely developers are to adopt it. The
47+
more developers adopt, the more tests are written. And the more tests are
48+
written, the better our lives as software engineers will be.
49+
50+
## Proposed solution
51+
52+
We propose defining and implementing a stable ABI for using Swift Testing that
53+
can be reliably adopted by various IDEs and other tools. There are two aspects
54+
of this ABI we need to implement:
55+
56+
- A stable entry point function that can be resolved dynamically at runtime (on
57+
platforms with dynamic loaders such as Darwin, Linux, and Windows.) This
58+
function needs a signature that will not change over time and which will take
59+
input and pass back asynchronous output in a format that a wide variety of
60+
tools will be able to interpret (whether they are written in Swift or not.)
61+
62+
This function should be implemented in Swift as it is expected to be used by
63+
code that can call into Swift, but which cannot rely on the specific binary
64+
minutiae of a given copy of Swift Testing.
65+
66+
- A stable format for input that can be passed to the entry point function and
67+
which can also be passed at the command line; and a stable format for output
68+
that can be consumed by tools to interpret test results.
69+
70+
Some tools cannot directly link to Swift code and must instead rely on
71+
command-line invocations of `swift test`. These tools will be able to pass
72+
their test configuration and options as an argument in the stable format and
73+
will be able to receive event information in the same stable format via a
74+
dedicated channel such as a file or named pipe.
75+
76+
> [!NOTE]
77+
> This document proposes defining a stable format for input and output, but only
78+
> actually defines the JSON schema for _output_. We intend to define the schema
79+
> for input in a subsequent proposal.
80+
>
81+
> In the interim, early adopters can encode an instance of Swift Testing's
82+
> `__CommandLineArguments_v0` type using `JSONEncoder`.
83+
84+
## Detailed design
85+
86+
We propose defining the stable input and output format using JSON as it is
87+
widely supported across platforms and languages. The proposed JSON schema for
88+
output is defined [here](../ABI/JSON.md).
89+
90+
### Invoking from the command line
91+
92+
When invoking `swift test`, we propose adding three new arguments to Swift
93+
Package Manager:
94+
95+
| Argument | Value Type | Description |
96+
|---|:-:|---|
97+
| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. |
98+
| `--event-stream-output` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. |
99+
| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. |
100+
101+
If `--configuration-path` is specified, Swift Testing will open it for reading
102+
and attempt to decode its contents as JSON. If `--event-stream-output` is
103+
specified, Swift Testing will open it for writing and will write a sequence of
104+
[JSON Lines](https://jsonlines.org) to it representing the data and events
105+
produced by the test run. `--event-stream-version` determines the stable schema
106+
used for output; pass `0` to match the schema proposed in this document.
107+
108+
> [!NOTE]
109+
> If `--event-stream-output` is specified but `--event-stream-version` is not,
110+
> the format _currently_ used is based on direct JSON encodings of the internal
111+
> Swift structures used by Swift Testing. This format is necessary to support
112+
> Xcode 16 Beta 1. In the future, the default value of this argument will be
113+
> assumed to be `0` instead (i.e. the JSON schema will match what we are
114+
> proposing here.)
115+
116+
On platforms that support them, callers can use a named pipe with
117+
`--event-stream-output` to get live results back from the test run rather than
118+
needing to wait until the file is closed by the test process. Named pipes can be
119+
created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html)
120+
function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew)
121+
function.
122+
123+
If `--configuration-path` is specified in addition to explicit command-line
124+
options like `--no-parallel`, the explicit command-line options take priority.
125+
126+
### Invoking from Swift
127+
128+
Tools that can link to and call Swift directly have the option of instantiating
129+
the tools-only SPI type `Runner`, however this is only possible if the tools and
130+
the test target link to the exact same copy of Swift Testing. To support tools
131+
that may link to a different copy (intentionally or otherwise), we propose
132+
adding an exported symbol to the Swift Testing library with the following Swift
133+
signature:
134+
135+
```swift
136+
@_spi(ForToolsIntegrationOnly)
137+
public enum ABIv0 {
138+
/* ... */
139+
140+
/// The type of the entry point to the testing library used by tools that want
141+
/// to remain version-agnostic regarding the testing library.
142+
///
143+
/// - Parameters:
144+
/// - configurationJSON: A buffer to memory representing the test
145+
/// configuration and options. If `nil`, a new instance is synthesized
146+
/// from the command-line arguments to the current process.
147+
/// - recordHandler: A JSON record handler to which is passed a buffer to
148+
/// memory representing each record as described in `ABI/JSON.md`.
149+
///
150+
/// - Returns: Whether or not the test run finished successfully.
151+
///
152+
/// - Throws: Any error that occurred prior to running tests. Errors that are
153+
/// thrown while tests are running are handled by the testing library.
154+
public typealias EntryPoint = @convention(thin) @Sendable (
155+
_ configurationJSON: UnsafeRawBufferPointer?,
156+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
157+
) async throws -> Bool
158+
159+
/// The entry point to the testing library used by tools that want to remain
160+
/// version-agnostic regarding the testing library.
161+
///
162+
/// The value of this property is a Swift function that can be used by tools
163+
/// that do not link directly to the testing library and wish to invoke tests
164+
/// in a binary that has been loaded into the current process. The value of
165+
/// this property is accessible from C and C++ as a function with name
166+
/// `"swt_abiv0_getEntryPoint"` and can be dynamically looked up at runtime
167+
/// using `dlsym()` or a platform equivalent.
168+
///
169+
/// The value of this property can be thought of as equivalent to
170+
/// `swift test --event-stream-output` except that, instead of streaming JSON
171+
/// records to a named pipe or file, it streams them to an in-process
172+
/// callback.
173+
public static var entryPoint: EntryPoint { get }
174+
}
175+
```
176+
177+
The inputs and outputs to this function are typed as `UnsafeRawBufferPointer`
178+
rather than `Data` because the latter is part of Foundation, and adding a public
179+
dependency on a Foundation type would make it very difficult for Foundation to
180+
adopt Swift Testing. It is a goal of the Swift Testing team to keep our Swift
181+
dependency list as small as possible.
182+
183+
We expect most tools that need to make use of this entry point will not be able
184+
to directly link to the exported Swift symbol and will instead need to look it
185+
up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html)
186+
or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress).
187+
The `ABIv0.entryPoint` property's getter will be exported to C and C++ as:
188+
189+
```c++
190+
extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void);
191+
```
192+
193+
The value returned from this C function is a direct representation of the value
194+
of `ABIv0.entryPoint` and can be cast back to its Swift function type using
195+
[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29).
196+
197+
On platforms where data-pointer-to-function-pointer conversion is disallowed per
198+
the C standard, this operation is unsupported. See §6.3.2.3 and §J.5.7 of
199+
[the C standard](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf).
200+
201+
> [!NOTE]
202+
> Swift Testing is statically linked into the main executable when it is
203+
> included as a package dependency. On Linux and other platforms that use the
204+
> ELF executable format, symbol information for the main executable may not be
205+
> available at runtime unless the `--export-dynamic` flag is passed to the
206+
> linker.
207+
208+
## Source compatibility
209+
210+
The changes proposed in this document are additive.
211+
212+
## Integration with supporting tools
213+
214+
Tools are able to use the proposed additions as described above.
215+
216+
## Future directions
217+
218+
- Extending the JSON schema to cover _input_ as well as _output_. As discussed,
219+
we will do so in a subsequent proposal.
220+
221+
- Extending the JSON schema to include richer information about events such as
222+
specific mismatched values in `#expect()` calls. This information is complex
223+
and we need to take care to model it efficiently and clearly.
224+
225+
- Adding Markdown or other formats to event messages. Rich text can be used by
226+
tools to emphasize values, switch to code voice, provide improved
227+
accessibility, etc.
228+
229+
- Adding additional entry points for different access patterns. We anticipate
230+
that a Swift function and a command-line interface are sufficient to cover
231+
most real-world use cases, but it may be the case that tools could use other
232+
mechanisms for starting test runs such as pure C or Objective-C interfaces,
233+
platform-specific interfaces, or bindings to other languages like Rust or Go.
234+
235+
## Alternatives considered
236+
237+
- Doing nothing. If we made no changes, we would be effectively requiring
238+
developers to use Xcode for all Swift Testing development and would be
239+
requiring third-party tools to parse human-readable command-line output. This
240+
approach would run counter to several of the Swift project's high-level goals
241+
and would not represent a true cross-platform solution.
242+
243+
- Using direct JSON encodings of Swift Testing's internal types to represent
244+
output. We initially attempted this and you can see the results in the Swift
245+
Testing repository if you look for "snapshot" types. A major downside became
246+
apparent quickly: these data types don't make for particularly usable JSON
247+
unless you're using `JSONDecoder` to convert back to them, and the default
248+
JSON encodings produced with `JSONEncoder` are not stable if we e.g. add
249+
enumeration cases with associated values or add non-optional fields to types.
250+
251+
- Using a format other than JSON. We considered using XML, YAML, Apple property
252+
lists, and a few other formats. JSON won out pretty quickly though: it is
253+
widely supported across platforms and languages and it is trivial to create
254+
Swift structures that encode to a well-designed JSON schema using
255+
`JSONEncoder`. Property lists would be just as easy to create, but it is a
256+
proprietary format and would not be trivially decodable on non-Apple platforms
257+
or using non-Apple tools.
258+
259+
- Exposing the C interface as a function that returns heap-allocated memory
260+
containing a Swift function reference. This allows us to emit a "thick" Swift
261+
function but requires callers to manually manage the resulting memory, and it
262+
may be difficult to reason about code that requires an extra level of pointer
263+
indirection. By having the C entry point function return a thin Swift function
264+
instead, the caller need only bitcast it and can call it directly, and the
265+
equivalent Swift interface can simply be a property getter rather than a
266+
function call.
267+
268+
## Acknowledgments
269+
270+
Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his
271+
tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand)
272+
for putting up with my incessant revisions and nitpicking while he worked on
273+
VS Code's Swift Testing support.
274+
275+
Thanks to the rest of the Swift Testing team for reviewing this proposal and the
276+
JSON schema and to the community for embracing Swift Testing!

‎Sources/Testing/EntryPoints/ABIEntryPoint.swift

Lines changed: 13 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -9,65 +9,27 @@
99
//
1010

1111
#if canImport(Foundation) && !SWT_NO_ABI_ENTRY_POINT
12-
/// The type of the entry point to the testing library used by tools that want
13-
/// to remain version-agnostic regarding the testing library.
14-
///
15-
/// - Parameters:
16-
/// - argumentsJSON: A buffer to memory representing the JSON encoding of an
17-
/// instance of `__CommandLineArguments_v0`. If `nil`, a new instance is
18-
/// created from the command-line arguments to the current process.
19-
/// - recordHandler: A JSON record handler to which is passed a buffer to
20-
/// memory representing each record as described in `ABI/JSON.md`.
21-
///
22-
/// - Returns: The result of invoking the testing library. The type of this
23-
/// value is subject to change.
24-
///
25-
/// - Throws: Any error that occurred prior to running tests. Errors that are
26-
/// thrown while tests are running are handled by the testing library.
27-
///
28-
/// This function examines the command-line arguments to the current process
29-
/// and then invokes available tests in the current process.
12+
private import _TestingInternals
13+
14+
/// An older signature for ``ABIv0/EntryPoint-swift.typealias`` used by Xcode 16
15+
/// Beta 1.
3016
///
31-
/// - Warning: This function's signature and the structure of its JSON inputs
32-
/// and outputs have not been finalized yet.
33-
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
34-
public typealias ABIEntryPoint_v0 = @Sendable (
17+
/// This type will be removed in a future update.
18+
@available(*, deprecated, message: "Use ABIv0.EntryPoint instead.")
19+
typealias ABIEntryPoint_v0 = @Sendable (
3520
_ argumentsJSON: UnsafeRawBufferPointer?,
3621
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
3722
) async throws -> CInt
3823

39-
/// Get the entry point to the testing library used by tools that want to remain
40-
/// version-agnostic regarding the testing library.
24+
/// An older signature for ``ABIv0/entryPoint-swift.type.property`` used by
25+
/// Xcode 16 Beta 1.
4126
///
42-
/// - Returns: A pointer to an instance of ``ABIEntryPoint_v0`` representing the
43-
/// ABI-stable entry point to the testing library. The caller owns this memory
44-
/// and is responsible for deinitializing and deallocating it when done.
45-
///
46-
/// This function can be used by tools that do not link directly to the testing
47-
/// library and wish to invoke tests in a binary that has been loaded into the
48-
/// current process. The function is emitted into the binary under the name
49-
/// `"swt_copyABIEntryPoint_v0"` and can be dynamically looked up at runtime
50-
/// using `dlsym()` or a platform equivalent.
51-
///
52-
/// The returned function can be thought of as equivalent to
53-
/// `swift test --experimental-event-stream-output` except that, instead of
54-
/// streaming JSON records to a named pipe or file, it streams them to an
55-
/// in-process callback.
56-
///
57-
/// - Warning: This function's signature and the structure of its JSON inputs
58-
/// and outputs have not been finalized yet.
27+
/// This function will be removed in a future update.
28+
@available(*, deprecated, message: "Use ABIv0.entryPoint (swt_abiv0_getEntryPoint()) instead.")
5929
@_cdecl("swt_copyABIEntryPoint_v0")
60-
@_spi(Experimental) @_spi(ForToolsIntegrationOnly)
61-
public func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
30+
@usableFromInline func copyABIEntryPoint_v0() -> UnsafeMutableRawPointer {
6231
let result = UnsafeMutablePointer<ABIEntryPoint_v0>.allocate(capacity: 1)
63-
result.initialize { argumentsJSON, recordHandler in
64-
let args = try argumentsJSON.map { argumentsJSON in
65-
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
66-
}
67-
68-
let eventHandler = try eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
69-
return await entryPoint(passing: args, eventHandler: eventHandler)
70-
}
32+
result.initialize { try await ABIv0.entryPoint($0, $1) ? EXIT_SUCCESS : EXIT_FAILURE }
7133
return .init(result)
7234
}
7335
#endif

0 commit comments

Comments
 (0)