Skip to content

Commit defa22b

Browse files
grynspangrynspan
committed
Propose a stabilized C-compatible ABI interface and JSON schema for event streaming.
1 parent 8edb08f commit defa22b

File tree

4 files changed

+369
-2
lines changed

4 files changed

+369
-2
lines changed

‎Documentation/Proposals/NNNN-json-abi.md

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
# A stable JSON-based ABI for tools integration
2+
3+
* Proposal: [SWT-NNNN](NNNN-json-abi.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Status: **Awaiting review**
6+
* Implementation: [apple/swift-testing#383](https://github.com/apple/swift-testing/pull/383),
7+
[apple/swift-testing#402](https://github.com/apple/swift-testing/pull/402)
8+
<!-- * Review: ([pitch](https://forums.swift.org/...)) -->
9+
10+
11+
## Introduction
12+
13+
One of the core components of Swift Testing is its ability to interoperate with
14+
Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced
15+
across all platforms supported by Swift, and can be added as a package
16+
dependency (or—eventually—linked from the Swift toolchain.)
17+
18+
## Motivation
19+
20+
Because Swift Testing may be used in various forms, and because integration with
21+
various tools is critical to its success, we need it to have a stable interface
22+
that can be used regardless of how it's been added to a package. There are a few
23+
patterns in particular we know we need to support:
24+
25+
- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing:
26+
the copy used by the IDE might be the same as the copy that tests use, in
27+
which case interoperation is trivial, but it may also be distinct if the tests
28+
use Swift Testing as a package dependency.
29+
30+
In the case of Xcode 16, Swift Testing is built as a framework much like
31+
XCTest and is automatically linked by test targets in an Xcode project or
32+
Swift package, but if the test target specifies a package dependency on Swift
33+
Testing, that dependency will take priority when the test code is compiled.
34+
35+
- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and
36+
perhaps, as with VS Code, cannot because it is not natively compiled): such an
37+
IDE needs a way to configure and invoke test code and then to read events back
38+
as they occur, but cannot touch the Swift symbols used by the tests.
39+
40+
In the case of VS Code, because it is implemented using TypeScript, it is not
41+
able to directly link to Swift Testing or other Swift libraries. In order for
42+
it to interpret events from a test run like "test passed" or "issue recorded",
43+
it needs to receive those events in a format it can understand.
44+
45+
Tools integration is important to the success of Swift Testing. The more tools
46+
provide integrations for it, the more likely developers are to adopt it. The
47+
more developers adopt, the more tests are written. And the more tests are
48+
written, the better our lives as software engineers will be.
49+
50+
## Proposed solution
51+
52+
We propose defining and implementing a stable ABI for using Swift Testing that
53+
can be reliably adopted by various IDEs and other tools. There are two aspects
54+
of this ABI we need to implement:
55+
56+
- A stable entry point function that can be resolved dynamically at runtime (on
57+
platforms with dynamic loaders such as Darwin, Linux, and Windows.) This
58+
function needs a signature that will not change over time and which will take
59+
input and pass back asynchronous output in a format that a wide variety of
60+
tools will be able to interpret (whether they are written in Swift or not.)
61+
62+
This function should be implemented in Swift as it is expected to be used by
63+
code that can call into Swift, but which cannot rely on the specific binary
64+
minutiae of a given copy of Swift Testing.
65+
66+
- A stable format for input and output that can be passed to the entry point
67+
function and which can also be passed at the command line for tools that
68+
cannot directly link to Swift code.
69+
70+
Tools that rely on command-line invocations of `swift test` can pass test
71+
configuration and options as an argument in the stable format and can receive
72+
event information in the same stable format via a dedicated channel such as a
73+
file or named pipe.
74+
75+
> [!NOTE]
76+
> This document proposes defining a stable format for input and output, but only
77+
> actually defines the JSON schema for _output_. We intend to define the schema
78+
> for input in a subsequent proposal.
79+
>
80+
> In the interim, early adopters can encode an instance of Swift Testing's
81+
> `__CommandLineArguments_v0` type using `JSONEncoder`.
82+
83+
## Detailed design
84+
85+
We propose defining the stable input and output format using JSON as it is
86+
widely supported across platforms and languages. The proposed JSON schema for
87+
output is defined [here](../ABI/JSON.md).
88+
89+
### Invoking from the command line
90+
91+
When invoking `swift test`, we propose adding three new arguments to Swift
92+
Package Manager:
93+
94+
| Argument | Value Type | Description |
95+
|---|:-:|---|
96+
| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. |
97+
| `--event-stream-output` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. |
98+
| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. |
99+
100+
If `--configuration-path` is specified, Swift Testing will open it for reading
101+
and attempt to decode its contents as JSON. If `--event-stream-output` is
102+
specified, Swift Testing will open it for writing and will write a sequence of
103+
[JSON Lines](https://jsonlines.org) to it representing the data and events
104+
produced by the test run. `--event-stream-version` determines the stable schema
105+
used for output; pass `0` to match the schema proposed in this document.
106+
107+
> [!NOTE]
108+
> If `--event-stream-output` is specified but `--event-stream-version` is not,
109+
> the format _currently_ used is based on direct JSON encodings of the internal
110+
> Swift structures used by Swift Testing. This format is necessary to support
111+
> Xcode 16 Beta 1. In the future, the default value of this argument will be
112+
> assumed to be `0` instead (i.e. the JSON schema will match what we are
113+
> proposing here.)
114+
115+
On platforms that support them, callers can use a named pipe with
116+
`--event-stream-output` to get live results back from the test run rather than
117+
needing to wait until the file is closed by the test process. Named pipes can be
118+
created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html)
119+
function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew)
120+
function.
121+
122+
If `--configuration-path` is specified in addition to explicit command-line
123+
options like `--no-parallel`, the explicit command-line options take priority.
124+
125+
### Invoking from Swift
126+
127+
For tools that can link to and call Swift directly, we propose adding an
128+
exported symbol to the Swift Testing library with the following Swift
129+
signature:
130+
131+
```swift
132+
@_spi(ForToolsIntegrationOnly)
133+
extension ABIv0 {
134+
/// The type of the entry point to the testing library used by tools that want
135+
/// to remain version-agnostic regarding the testing library.
136+
///
137+
/// - Parameters:
138+
/// - configurationJSON: A buffer to memory representing the test
139+
/// configuration and options. If `nil`, a new instance is synthesized
140+
/// from the command-line arguments to the current process.
141+
/// - recordHandler: A JSON record handler to which is passed a buffer to
142+
/// memory representing each record as described in `ABI/JSON.md`.
143+
///
144+
/// - Returns: Whether or not the test run finished successfully.
145+
///
146+
/// - Throws: Any error that occurred prior to running tests. Errors that are
147+
/// thrown while tests are running are handled by the testing library.
148+
public typealias EntryPoint = @convention(thin) @Sendable (
149+
_ configurationJSON: UnsafeRawBufferPointer?,
150+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
151+
) async throws -> Bool
152+
153+
/// The entry point to the testing library used by tools that want to remain
154+
/// version-agnostic regarding the testing library.
155+
///
156+
/// The value of this property is a Swift function that can be used by tools
157+
/// that do not link directly to the testing library and wish to invoke tests
158+
/// in a binary that has been loaded into the current process. The value of
159+
/// this property is accessible from C and C++ as a function with name
160+
/// `"swt_ABIv0_copyEntryPoint"` and can be dynamically looked up at runtime
161+
/// using `dlsym()` or a platform equivalent.
162+
///
163+
/// The value of this property can be thought of as equivalent to
164+
/// `swift test --event-stream-output` except that, instead of streaming JSON
165+
/// records to a named pipe or file, it streams them to an in-process
166+
/// callback.
167+
public static var entryPoint: EntryPoint { get }
168+
}
169+
```
170+
171+
We expect most tools that need to make use of this entry point will not be able
172+
to directly link to the exported Swift symbol and will instead need to look it
173+
up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html)
174+
or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress).
175+
The same function will be exported to C and C++ as:
176+
177+
```c++
178+
extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void);
179+
```
180+
181+
The value returned from this C function is a direct representation of the value
182+
of `ABIv0.entryPoint` and can be cast back to its Swift function type using
183+
[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29).
184+
185+
> [!NOTE]
186+
> On Linux and other platforms that use the ELF executable format, symbol
187+
> information for the main executable may not be available at runtime unless it
188+
> has been linked with the `--export-dynamic` flag.
189+
190+
## Source compatibility
191+
192+
The changes proposed in this document are additive.
193+
194+
## Integration with supporting tools
195+
196+
Tools are able to use the proposed additions as described above.
197+
198+
## Future directions
199+
200+
- Extending the JSON schema to cover _input_ as well as _output_. As discussed,
201+
we will do so in a subsequent proposal.
202+
203+
- Extending the JSON schema to include richer information about events such as
204+
specific mismatched values in `#expect()` calls. This information is complex
205+
and we need to take care to model it efficiently and clearly.
206+
207+
- Adding Markdown or other formats to event messages. Rich text can be used by
208+
tools to emphasize values, switch to code voice, provide improved
209+
accessibility, etc.
210+
211+
- Adding additional entry points for different access patterns. We anticipate
212+
that a Swift function and a command-line interface are sufficient to cover
213+
most real-world use cases, but it may be the case that tools could use other
214+
mechanisms for starting test runs such as pure C or Objective-C interfaces,
215+
platform-specific interfaces, or bindings to other languages like Rust or Go.
216+
217+
## Alternatives considered
218+
219+
- Doing nothing. If we made no changes, we would be effectively requiring
220+
developers to use Xcode for all Swift Testing development and would be
221+
requiring third-party tools to parse human-readable command-line output. This
222+
approach would run counter to several of the Swift project's high-level goals
223+
and would not represent a true cross-platform solution.
224+
225+
- Using direct JSON encodings of Swift Testing's internal types to represent
226+
output. We initially attempted this and you can see the results in the Swift
227+
Testing repository if you look for "snapshot" types. A major downside became
228+
apparent quickly: these data types don't make for particularly usable JSON
229+
unless you're using `JSONDecoder` to convert back to them, and the default
230+
JSON encodings produced with `JSONEncoder` are not stable if we e.g. add
231+
enumeration cases with associated values or add non-optional fields to types.
232+
233+
- Using a format other than JSON. We considered using XML, YAML, Apple property
234+
lists, and a few other formats. JSON won out pretty quickly though: it is
235+
widely supported across platforms and languages and it is trivial to create
236+
Swift structures that encode to a well-designed JSON schema using
237+
`JSONEncoder`. Property lists would be just as easy to create, but it is a
238+
proprietary format and would not be trivially decodable on non-Apple platforms
239+
or using non-Apple tools.
240+
241+
- Exposing the C interface as a function that returns heap-allocated memory
242+
containing a Swift function reference. This allows us to emit a "thick" Swift
243+
function but requires callers to manually manage the resulting memory, and it
244+
may be difficult to reason about code that requires an extra level of pointer
245+
indirection. By having the C entry point function return a thin Swift function
246+
instead, the caller need only bitcast it and can call it directly, and the
247+
equivalent Swift interface can simply be a property getter rather than a
248+
function call.
249+
250+
## Acknowledgments
251+
252+
Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his
253+
tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand)
254+
for putting up with my incessant revisions and nitpicking while he worked on
255+
VS Code's Swift Testing support.
256+
257+
Thanks to the rest of the Swift Testing team for reviewing this proposal and the
258+
JSON schema and to the community for embracing Swift Testing!

‎Sources/Testing/EntryPoints/ABIv0/ABIv0.EntryPoint.swift

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2023 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
#if canImport(Foundation) && !SWT_NO_ABI_ENTRY_POINT
12+
private import _TestingInternals
13+
14+
@_spi(ForToolsIntegrationOnly)
15+
extension ABIv0 {
16+
/// The type of the entry point to the testing library used by tools that want
17+
/// to remain version-agnostic regarding the testing library.
18+
///
19+
/// - Parameters:
20+
/// - configurationJSON: A buffer to memory representing the test
21+
/// configuration and options. If `nil`, a new instance is synthesized
22+
/// from the command-line arguments to the current process.
23+
/// - recordHandler: A JSON record handler to which is passed a buffer to
24+
/// memory representing each record as described in `ABI/JSON.md`.
25+
///
26+
/// - Returns: Whether or not the test run finished successfully.
27+
///
28+
/// - Throws: Any error that occurred prior to running tests. Errors that are
29+
/// thrown while tests are running are handled by the testing library.
30+
public typealias EntryPoint = @convention(thin) @Sendable (
31+
_ configurationJSON: UnsafeRawBufferPointer?,
32+
_ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void
33+
) async throws -> Bool
34+
35+
/// The entry point to the testing library used by tools that want to remain
36+
/// version-agnostic regarding the testing library.
37+
///
38+
/// The value of this property is a Swift function that can be used by tools
39+
/// that do not link directly to the testing library and wish to invoke tests
40+
/// in a binary that has been loaded into the current process. The value of
41+
/// this property is accessible from C and C++ as a function with name
42+
/// `"swt_ABIv0_copyEntryPoint"` and can be dynamically looked up at runtime
43+
/// using `dlsym()` or a platform equivalent.
44+
///
45+
/// The value of this property can be thought of as equivalent to
46+
/// `swift test --event-stream-output` except that, instead of streaming JSON
47+
/// records to a named pipe or file, it streams them to an in-process
48+
/// callback.
49+
public static var entryPoint: EntryPoint {
50+
return { argumentsJSON, recordHandler in
51+
let args = try argumentsJSON.map { argumentsJSON in
52+
try JSON.decode(__CommandLineArguments_v0.self, from: argumentsJSON)
53+
}
54+
55+
let eventHandler = try eventHandlerForStreamingEvents(version: args?.experimentalEventStreamVersion, forwardingTo: recordHandler)
56+
let exitCode = await Testing.entryPoint(passing: args, eventHandler: eventHandler)
57+
return exitCode == EXIT_SUCCESS
58+
}
59+
}
60+
}
61+
62+
/// An exported C function that is the equivalent of ``ABIv0/entryPoint``.
63+
///
64+
/// - Returns: The value of ``ABIv0/entryPoint`` cast to an untyped pointer.
65+
@_cdecl("swt_abiv0_getEntryPoint")
66+
@usableFromInline func abiv0_getEntryPoint() -> UnsafeRawPointer {
67+
unsafeBitCast(ABIv0.entryPoint, to: UnsafeRawPointer.self)
68+
}
69+
#endif

‎Sources/Testing/EntryPoints/ABIv0/ABIv0.swift

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@
99
//
1010

1111
/// A namespace for ABI version 0 symbols.
12-
enum ABIv0: Sendable {}
12+
@_spi(ForToolsIntegrationOnly)
13+
public enum ABIv0: Sendable {}

‎Tests/TestingTests/ABIEntryPointTests.swift

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ private import _TestingInternals
1919

2020
@Suite("ABI entry point tests")
2121
struct ABIEntryPointTests {
22-
@Test func v0() async throws {
22+
@Test func v0_experimental() async throws {
2323
#if !os(Linux) && !SWT_NO_DYNAMIC_LINKING
2424
// Get the ABI entry point by dynamically looking it up at runtime.
2525
//
@@ -61,5 +61,44 @@ struct ABIEntryPointTests {
6161
// Validate expectations.
6262
#expect(result == EXIT_SUCCESS)
6363
}
64+
65+
@Test func v0() async throws {
66+
#if !os(Linux) && !SWT_NO_DYNAMIC_LINKING
67+
// Get the ABI entry point by dynamically looking it up at runtime.
68+
//
69+
// NOTE: The standard Linux linker does not allow exporting symbols from
70+
// executables, so dlsym() does not let us find this function on that
71+
// platform when built as an executable rather than a dynamic library.
72+
let abiv0_getEntryPoint = try #require(
73+
symbol(named: "swt_abiv0_getEntryPoint").map {
74+
unsafeBitCast($0, to: (@convention(c) () -> UnsafeRawPointer).self)
75+
}
76+
)
77+
#endif
78+
let abiEntryPoint = unsafeBitCast(abiv0_getEntryPoint(), to: ABIv0.EntryPoint.self)
79+
80+
// Construct arguments and convert them to JSON.
81+
var arguments = __CommandLineArguments_v0()
82+
arguments.filter = ["NonExistentTestThatMatchesNothingHopefully"]
83+
arguments.experimentalEventStreamVersion = 0
84+
arguments.verbosity = .min
85+
let argumentsJSON = try JSON.withEncoding(of: arguments) { argumentsJSON in
86+
let result = UnsafeMutableRawBufferPointer.allocate(byteCount: argumentsJSON.count, alignment: 1)
87+
result.copyMemory(from: argumentsJSON)
88+
return result
89+
}
90+
defer {
91+
argumentsJSON.deallocate()
92+
}
93+
94+
// Call the entry point function.
95+
let result = try await abiEntryPoint(.init(argumentsJSON)) { recordJSON in
96+
let record = try! JSON.decode(ABIv0.Record.self, from: recordJSON)
97+
_ = record.version
98+
}
99+
100+
// Validate expectations.
101+
#expect(result)
102+
}
64103
}
65104
#endif

0 commit comments

Comments
 (0)