Skip to content

Commit 5dc96fd

Browse files
grynspangrynspan
committed
Propose a stabilized C-compatible ABI interface and JSON schema for event streaming.
1 parent 8edb08f commit 5dc96fd

File tree

4 files changed

+384
-2
lines changed

4 files changed

+384
-2
lines changed

‎Documentation/Proposals/NNNN-json-abi.md

Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
# A stable JSON-based ABI for tools integration
2+
3+
* Proposal: [SWT-NNNN](NNNN-json-abi.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Status: **Awaiting review**
6+
* Implementation: [apple/swift-testing#383](https://github.com/apple/swift-testing/pull/383),
7+
[apple/swift-testing#402](https://github.com/apple/swift-testing/pull/402)
8+
<!-- * Review: ([pitch](https://forums.swift.org/...)) -->
9+
10+
11+
## Introduction
12+
13+
One of the core components of Swift Testing is its ability to interoperate with
14+
Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced
15+
across all platforms supported by Swift, and can be added as a package
16+
dependency (or—eventually—linked from the Swift toolchain.)
17+
18+
## Motivation
19+
20+
Because Swift Testing may be used in various forms, and because integration with
21+
various tools is critical to its success, we need it to have a stable interface
22+
that can be used regardless of how it's been added to a package. There are a few
23+
patterns in particular we know we need to support:
24+
25+
- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing:
26+
the copy used by the IDE might be the same as the copy that tests use, in
27+
which case interoperation is trivial, but it may also be distinct if the tests
28+
use Swift Testing as a package dependency.
29+
30+
In the case of Xcode 16, Swift Testing is built as a framework much like
31+
XCTest and is automatically linked by test targets in an Xcode project or
32+
Swift package, but if the test target specifies a package dependency on Swift
33+
Testing, that dependency will take priority when the test code is compiled.
34+
35+
- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and
36+
perhaps, as with VS Code, cannot because it is not natively compiled): such an
37+
IDE needs a way to configure and invoke test code and then to read events back
38+
as they occur, but cannot touch the Swift symbols used by the tests.
39+
40+
In the case of VS Code, because it is implemented using TypeScript, it is not
41+
able to directly link to Swift Testing or other Swift libraries. In order for
42+
it to interpret events from a test run like "test passed" or "issue recorded",
43+
it needs to receive those events in a format it can understand.
44+
45+
Tools integration is important to the success of Swift Testing. The more tools
46+
provide integrations for it, the more likely developers are to adopt it. The
47+
more developers adopt, the more tests are written. And the more tests are
48+
written, the better our lives as software engineers will be.
49+
50+
## Proposed solution
51+
52+
We propose defining and implementing a stable ABI for using Swift Testing that
53+
can be reliably adopted by various IDEs and other tools. There are two aspects
54+
of this ABI we need to implement:
55+
56+
- A stable entry point function that can be resolved dynamically at runtime (on
57+
platforms with dynamic loaders such as Darwin, Linux, and Windows.) This
58+
function needs a signature that will not change over time and which will take
59+
input and pass back asynchronous output in a format that a wide variety of
60+
tools will be able to interpret (whether they are written in Swift or not.)
61+
62+
This function should be implemented in Swift as it is expected to be used by
63+
code that can call into Swift, but which cannot rely on the specific binary
64+
minutiae of a given copy of Swift Testing.
65+
66+
- A stable format for input and output that can be passed to the entry point
67+
function and which can also be passed at the command line for tools that
68+
cannot directly link to Swift code.
69+
70+
Tools that rely on command-line invocations of `swift test` can pass test
71+
configuration and options as an argument in the stable format and can receive
72+
event information in the same stable format via a dedicated channel such as a
73+
file or named pipe.
74+
75+
> [!NOTE]
76+
> This document proposes defining a stable format for input and output, but only
77+
> actually defines the JSON schema for _output_. We intend to define the schema
78+
> for input in a subsequent proposal.
79+
>
80+
> In the interim, early adopters can encode an instance of Swift Testing's
81+
> `__CommandLineArguments_v0` type using `JSONEncoder`.
82+
83+
## Detailed design
84+
85+
We propose defining the stable input and output format using JSON as it is
86+
widely supported across platforms and languages. The proposed JSON schema for
87+
output is defined [here](../ABI/JSON.md).
88+
89+
### Invoking from the command line
90+
91+
When invoking `swift test`, we propose adding three new arguments to Swift
92+
Package Manager:
93+
94+
| Argument | Value Type | Description |
95+
|---|:-:|---|
96+
| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. |
97+
| `--event-stream-output` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. |
98+
| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. |
99+
100+
If `--configuration-path` is specified, Swift Testing will open it for reading
101+
and attempt to decode its contents as JSON. If `--event-stream-output` is
102+
specified, Swift Testing will open it for writing and will write a sequence of
103+
[JSON Lines](https://jsonlines.org) to it representing the data and events
104+
produced by the test run. `--event-stream-version` determines the stable schema
105+
used for output; pass `0` to match the schema proposed in this document.
106+
107+
> [!NOTE]
108+
> If `--event-stream-output` is specified but `--event-stream-version` is not,
109+
> the format _currently_ used is based on direct JSON encodings of the internal
110+
> Swift structures used by Swift Testing. This format is necessary to support
111+
> Xcode 16 Beta 1. In the future, the default value of this argument will be
112+
> assumed to be `0` instead (i.e. the JSON schema will match what we are
113+
> proposing here.)
114+
115+
On platforms that support them, callers can use a named pipe with
116+
`--event-stream-output` to get live results back from the test run rather than
117+
needing to wait until the file is closed by the test process. Named pipes can be
118+
created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html)
119+
function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew)
120+
function.
121+
122+
If `--configuration-path` is specified in addition to explicit command-line
123+
options like `--no-parallel`, the explicit command-line options take priority.
124+
125+
### Invoking from Swift
126+
127+
Tools that can link to and call Swift directly have the option of instantiating
128+
the tools-only SPI type `Runner`, however this is only possible if the tools and
129+
the test target link to the exact same copy of Swift Testing. To support tools
130+
that may link to a different copy (intentionally or otherwise), we propose
131+
adding an exported symbol to the Swift Testing library with the following Swift
132+
signature:
133+
134+
```swift
135+
@_spi(ForToolsIntegrationOnly)
136+
extension ABIv0 {
137+
/// The type of the entry point to the testing library used by tools that want
138+
/// to remain version-agnostic regarding the testing library.
139+
///
140+
/// - Parameters:
141+
/// - configurationJSON: A buffer to memory representing the test
142+
/// configuration and options. If `nil`, a new instance is synthesized
143+
/// from the command-line arguments to the current process.
144+
/// - recordHandler: A JSON record handler to which is passed a buffer to
145+
/// memory representing each record as described in `ABI/JSON.md`.
146+
///
147+
/// - Returns: Whether or not the test run finished successfully.
148+
///
149+
/// - Throws: Any error that occurred prior to running tests. Errors that are
150+
/// thrown while tests are running are handled by the testing library.
151+
public typealias EntryPoint = @convention(thin) @Sendable (
152+
_ configurationJSON: UnsafeRawBufferPointer?,
153+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
154+
) async throws -> Bool
155+
156+
/// The entry point to the testing library used by tools that want to remain
157+
/// version-agnostic regarding the testing library.
158+
///
159+
/// The value of this property is a Swift function that can be used by tools
160+
/// that do not link directly to the testing library and wish to invoke tests
161+
/// in a binary that has been loaded into the current process. The value of
162+
/// this property is accessible from C and C++ as a function with name
163+
/// `"swt_ABIv0_copyEntryPoint"` and can be dynamically looked up at runtime
164+
/// using `dlsym()` or a platform equivalent.
165+
///
166+
/// The value of this property can be thought of as equivalent to
167+
/// `swift test --event-stream-output` except that, instead of streaming JSON
168+
/// records to a named pipe or file, it streams them to an in-process
169+
/// callback.
170+
public static var entryPoint: EntryPoint { get }
171+
}
172+
```
173+
174+
The inputs and outputs to this function are typed as `UnsafeRawBufferPointer`
175+
rather than `Data` because the latter is part of Foundation, and adding a public
176+
dependency on a Foundation type would make it very difficult for Foundation to
177+
adopt Swift Testing. It is a goal of the Swift Testing team to keep our Swift
178+
dependency list as small as possible.
179+
180+
We expect most tools that need to make use of this entry point will not be able
181+
to directly link to the exported Swift symbol and will instead need to look it
182+
up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html)
183+
or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress).
184+
The `ABIv0.entryPoint` property's getter will be exported to C and C++ as:
185+
186+
```c++
187+
extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void);
188+
```
189+
190+
The value returned from this C function is a direct representation of the value
191+
of `ABIv0.entryPoint` and can be cast back to its Swift function type using
192+
[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29).
193+
194+
On platforms where data-pointer-to-function-pointer conversion is disallowed per
195+
the C standard, this operation is unsupported. See §6.3.2.3 and §J.5.7 of
196+
[the C standard](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf).
197+
198+
> [!NOTE]
199+
> Swift Testing is statically linked into the main executable when it is
200+
> included as a package dependency. On Linux and other platforms that use the
201+
> ELF executable format, symbol information for the main executable may not be
202+
> available at runtime unless the `--export-dynamic` flag is passed to the
203+
> linker.
204+
205+
## Source compatibility
206+
207+
The changes proposed in this document are additive.
208+
209+
## Integration with supporting tools
210+
211+
Tools are able to use the proposed additions as described above.
212+
213+
## Future directions
214+
215+
- Extending the JSON schema to cover _input_ as well as _output_. As discussed,
216+
we will do so in a subsequent proposal.
217+
218+
- Extending the JSON schema to include richer information about events such as
219+
specific mismatched values in `#expect()` calls. This information is complex
220+
and we need to take care to model it efficiently and clearly.
221+
222+
- Adding Markdown or other formats to event messages. Rich text can be used by
223+
tools to emphasize values, switch to code voice, provide improved
224+
accessibility, etc.
225+
226+
- Adding additional entry points for different access patterns. We anticipate
227+
that a Swift function and a command-line interface are sufficient to cover
228+
most real-world use cases, but it may be the case that tools could use other
229+
mechanisms for starting test runs such as pure C or Objective-C interfaces,
230+
platform-specific interfaces, or bindings to other languages like Rust or Go.
231+
232+
## Alternatives considered
233+
234+
- Doing nothing. If we made no changes, we would be effectively requiring
235+
developers to use Xcode for all Swift Testing development and would be
236+
requiring third-party tools to parse human-readable command-line output. This
237+
approach would run counter to several of the Swift project's high-level goals
238+
and would not represent a true cross-platform solution.
239+
240+
- Using direct JSON encodings of Swift Testing's internal types to represent
241+
output. We initially attempted this and you can see the results in the Swift
242+
Testing repository if you look for "snapshot" types. A major downside became
243+
apparent quickly: these data types don't make for particularly usable JSON
244+
unless you're using `JSONDecoder` to convert back to them, and the default
245+
JSON encodings produced with `JSONEncoder` are not stable if we e.g. add
246+
enumeration cases with associated values or add non-optional fields to types.
247+
248+
- Using a format other than JSON. We considered using XML, YAML, Apple property
249+
lists, and a few other formats. JSON won out pretty quickly though: it is
250+
widely supported across platforms and languages and it is trivial to create
251+
Swift structures that encode to a well-designed JSON schema using
252+
`JSONEncoder`. Property lists would be just as easy to create, but it is a
253+
proprietary format and would not be trivially decodable on non-Apple platforms
254+
or using non-Apple tools.
255+
256+
- Exposing the C interface as a function that returns heap-allocated memory
257+
containing a Swift function reference. This allows us to emit a "thick" Swift
258+
function but requires callers to manually manage the resulting memory, and it
259+
may be difficult to reason about code that requires an extra level of pointer
260+
indirection. By having the C entry point function return a thin Swift function
261+
instead, the caller need only bitcast it and can call it directly, and the
262+
equivalent Swift interface can simply be a property getter rather than a
263+
function call.
264+
265+
## Acknowledgments
266+
267+
Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his
268+
tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand)
269+
for putting up with my incessant revisions and nitpicking while he worked on
270+
VS Code's Swift Testing support.
271+
272+
Thanks to the rest of the Swift Testing team for reviewing this proposal and the
273+
JSON schema and to the community for embracing Swift Testing!

‎Sources/Testing/EntryPoints/ABIv0/ABIv0.EntryPoint.swift

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2023 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
#if canImport(Foundation) && !SWT_NO_ABI_ENTRY_POINT
12+
private import _TestingInternals
13+
14+
@_spi(ForToolsIntegrationOnly)
15+
extension ABIv0 {
16+
/// The type of the entry point to the testing library used by tools that want
17+
/// to remain version-agnostic regarding the testing library.
18+
///
19+
/// - Parameters:
20+
/// - configurationJSON: A buffer to memory representing the test
21+
/// configuration and options. If `nil`, a new instance is synthesized
22+
/// from the command-line arguments to the current process.
23+
/// - recordHandler: A JSON record handler to which is passed a buffer to
24+
/// memory representing each record as described in `ABI/JSON.md`.
25+
///
26+
/// - Returns: Whether or not the test run finished successfully.
27+
///
28+
/// - Throws: Any error that occurred prior to running tests. Errors that are
29+
/// thrown while tests are running are handled by the testing library.
30+
public typealias EntryPoint = @convention(thin) @Sendable (
31+
_ configurationJSON: UnsafeRawBufferPointer?,
32+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
33+
) async throws -> Bool
34+
35+
/// The entry point to the testing library used by tools that want to remain
36+
/// version-agnostic regarding the testing library.
37+
///
38+
/// The value of this property is a Swift function that can be used by tools
39+
/// that do not link directly to the testing library and wish to invoke tests
40+
/// in a binary that has been loaded into the current process. The value of
41+
/// this property is accessible from C and C++ as a function with name
42+
/// `"swt_ABIv0_copyEntryPoint"` and can be dynamically looked up at runtime
43+
/// using `dlsym()` or a platform equivalent.
44+
///
45+
/// The value of this property can be thought of as equivalent to
46+
/// `swift test --event-stream-output` except that, instead of streaming JSON
47+
/// records to a named pipe or file, it streams them to an in-process
48+
/// callback.
49+
public static var entryPoint: EntryPoint {
50+
return { argumentsJSON, recordHandler in
51+
let args = try argumentsJSON.map { argumentsJSON in
52+
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
53+
}
54+
55+
let eventHandler = try eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
56+
let exitCode = await Testing.entryPoint(passing: args, eventHandler: eventHandler)
57+
return exitCode == EXIT_SUCCESS
58+
}
59+
}
60+
}
61+
62+
/// An exported C function that is the equivalent of ``ABIv0/entryPoint``.
63+
///
64+
/// - Returns: The value of ``ABIv0/entryPoint`` cast to an untyped pointer.
65+
@_cdecl("swt_abiv0_getEntryPoint")
66+
@usableFromInline func abiv0_getEntryPoint() -> UnsafeRawPointer {
67+
unsafeBitCast(ABIv0.entryPoint, to: UnsafeRawPointer.self)
68+
}
69+
#endif

‎Sources/Testing/EntryPoints/ABIv0/ABIv0.swift

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@
99
//
1010

1111
/// A namespace for ABI version 0 symbols.
12-
enum ABIv0: Sendable {}
12+
@_spi(ForToolsIntegrationOnly)
13+
public enum ABIv0: Sendable {}

0 commit comments

Comments
 (0)