[SWT-0002] A stable JSON-based ABI for tools integration #479

grynspan · 2024-06-13T13:45:32Z

One of the core components of Swift Testing is its ability to interoperate with Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced across all platforms supported by Swift, and can be added as a package dependency (or—eventually—linked from the Swift toolchain.)

Because Swift Testing may be used in various forms, and because integration with various tools is critical to its success, we need it to have a stable interface that can be used regardless of how it's been added to a package.

Read the full proposal here.

Checklist:

Code and documentation should follow the style of the Style Guide.
If public symbols are renamed or modified, DocC references should be updated.

grynspan · 2024-06-13T14:17:39Z

@younata This may be of interest to you as a tools author. :)

plemarquand

Overall this looks great, and I think satisfies the needs of 3rd party tools in a way that can evolve gracefully if need be.

Documentation/Proposals/NNNN-json-abi.md

grynspan · 2024-06-13T17:56:28Z

@swift-ci please test

grynspan · 2024-06-20T15:26:16Z

@swift-ci please test

@allevato

This PR changes the behavior of `swift test list` and its various synonyms to allow reporting the list of tests via the ABI-stable JSON mechanism described in #479. As it is not currently possible to directly call `swift test list --experimental-event-stream-output ... --experimental-event-stream-version 0`, it's a bit hard to test this code. However, it is possible to opt into this mode using `--experimental-configuration-path` and passing a path to a JSON file that includes `"listTests": true` (as noted by @allevato.) Resolves #506. Resolves rdar://130627856.

This PR changes the default JSON schema version for event stream output to version 0 (the version documented in #479.) Previously, if a caller did not specify a version number for the event stream, it would default to encoding "snapshots" of various values from the testing library. Xcode 16 Beta 1 uses this legacy format, so to continue supporting newer betas of Xcode 16 transitionally, this PR temporarily enables a "-1" JSON schema version that continues to emit snapshot encodings. This JSON schema version will be removed (along with the snapshots code in general) in a future update once Xcode has migrated. Resolves #527.

grynspan · 2024-07-10T13:06:33Z

Rebased on main.

grynspan · 2024-07-10T13:06:38Z

@swift-ci please test

@allevato

This PR changes the behavior of `swift test list` and its various synonyms to allow reporting the list of tests via the ABI-stable JSON mechanism described in #479. As it is not currently possible to directly call `swift test list --experimental-event-stream-output ... --experimental-event-stream-version 0`, it's a bit hard to test this code. However, it is possible to opt into this mode using `--experimental-configuration-path` and passing a path to a JSON file that includes `"listTests": true` (as noted by @allevato.) Resolves #506. Resolves rdar://130627856.

This PR changes the default JSON schema version for event stream output to version 0 (the version documented in #479.) Previously, if a caller did not specify a version number for the event stream, it would default to encoding "snapshots" of various values from the testing library. Xcode 16 Beta 1 uses this legacy format, so to continue supporting newer betas of Xcode 16 transitionally, this PR temporarily enables a "-1" JSON schema version that continues to emit snapshot encodings. This JSON schema version will be removed (along with the snapshots code in general) in a future update once Xcode has migrated. Resolves #527. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

…vent streaming.

… event stream version

grynspan · 2024-07-10T16:48:55Z

@swift-ci please test

grynspan · 2024-07-10T20:56:26Z

@swift-ci please test

grynspan · 2024-07-10T21:06:31Z

CI was passing prior to the last commit, which was documentation only. Merging.

@allevato

This PR changes the behavior of `swift test list` and its various synonyms to allow reporting the list of tests via the ABI-stable JSON mechanism described in #479. As it is not currently possible to directly call `swift test list --experimental-event-stream-output ... --experimental-event-stream-version 0`, it's a bit hard to test this code. However, it is possible to opt into this mode using `--experimental-configuration-path` and passing a path to a JSON file that includes `"listTests": true` (as noted by @allevato.) Resolves #506. Resolves rdar://130627856.

@allevato

This PR changes the behavior of `swift test list` and its various synonyms to allow reporting the list of tests via the ABI-stable JSON mechanism described in #479. As it is not currently possible to directly call `swift test list --event-stream-output-path ... --event-stream-version 0`, it's a bit hard to test this code. However, it is possible to opt into this mode using `--configuration-path` and passing a path to a JSON file that includes `"listTests": true` (as noted by @allevato.) Support for these arguments with `swift test list` is tracked by swiftlang/swift-package-manager#7768. Resolves #506. Resolves rdar://130627856. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

Now that we've landed and tagged #479 in 0.11.0, we no longer need to keep building the snapshot types for non-Darwin platforms because we have a stable, permanent replacement for them. We need to keep them around on Darwin to support Xcode 16 betas.

Now that we've landed and tagged #479 in 0.11.0, we no longer need to keep building the snapshot types for non-Darwin platforms because we have a stable, permanent replacement for them. We need to keep them around on Darwin to support Xcode 16 betas. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.

grynspan added enhancement New feature or request swiftpm-integration 📦 Swift Package Manager integration tools integration 🛠️ Integration of swift-testing into tools/IDEs public-api Affects public API api-proposal API proposal PRs (documentation only) labels Jun 13, 2024

grynspan self-assigned this Jun 13, 2024

grynspan requested a review from plemarquand June 13, 2024 14:00

grynspan force-pushed the jgrynspan/stable-json-pitch branch from defa22b to 5dc96fd Compare June 13, 2024 14:15

plemarquand approved these changes Jun 13, 2024

View reviewed changes

Documentation/Proposals/NNNN-json-abi.md Outdated Show resolved Hide resolved

Documentation/Proposals/NNNN-json-abi.md Outdated Show resolved Hide resolved

grynspan force-pushed the jgrynspan/stable-json-pitch branch 6 times, most recently from dac24fb to e7f4c5f Compare June 13, 2024 17:56

grynspan force-pushed the jgrynspan/stable-json-pitch branch from e7f4c5f to 2df15d2 Compare June 13, 2024 18:26

grynspan mentioned this pull request Jun 13, 2024

Don't emit JSON blobs for test cases if a test function isn't parameterized. #482

Merged

2 tasks

grynspan force-pushed the jgrynspan/stable-json-pitch branch 4 times, most recently from 9267638 to dd8988e Compare June 20, 2024 15:25

grynspan marked this pull request as ready for review June 20, 2024 15:26

grynspan requested review from stmontgomery, dennisweissmann, briancroom, SeanROlszewski and suzannaratcliff as code owners June 20, 2024 15:26

grynspan force-pushed the jgrynspan/stable-json-pitch branch from 9d16cad to 258d2ce Compare July 10, 2024 12:52

grynspan force-pushed the jgrynspan/stable-json-pitch branch from 258d2ce to e2c41be Compare July 10, 2024 13:06

grynspan requested a review from stmontgomery July 10, 2024 15:56

grynspan added 6 commits July 10, 2024 12:06

Propose a stabilized C-compatible ABI interface and JSON schema for e…

1bc6f94

…vent streaming.

Reorganize ABI files, put ABI entry point symbols in one file

b405e35

Remove redundant SPI attribute

dba93f0

Incorporate feedback

406e98d

Change argument name and clarify what happens if you don't specify an…

392e42c

… event stream version

Preserve snapshot behaviour for Xcode's older entry point function

006545f

grynspan force-pushed the jgrynspan/stable-json-pitch branch from b001bd8 to 006545f Compare July 10, 2024 16:48

Acceptance

fc847a8

grynspan changed the title ~~[SWT-NNNN] A stable JSON-based ABI for tools integration~~ [SWT-0002] A stable JSON-based ABI for tools integration Jul 10, 2024

grynspan merged commit 1b7dba9 into main Jul 10, 2024
1 of 3 checks passed

grynspan deleted the jgrynspan/stable-json-pitch branch July 10, 2024 21:06

grynspan mentioned this pull request Jul 10, 2024

Add the formalized Swift Testing event stream arguments to swift test. swiftlang/swift-package-manager#7768

Merged

grynspan mentioned this pull request Jul 12, 2024

[6.0] Add the formalized Swift Testing event stream arguments to swift test. swiftlang/swift-package-manager#7770

Merged

grynspan mentioned this pull request Jul 16, 2024

Remove Snapshot types from non-Darwin builds. #540

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SWT-0002] A stable JSON-based ABI for tools integration #479

[SWT-0002] A stable JSON-based ABI for tools integration #479

Uh oh!

grynspan commented Jun 13, 2024 •

edited

Loading

Uh oh!

grynspan commented Jun 13, 2024

Uh oh!

plemarquand left a comment

Uh oh!

Uh oh!

Uh oh!

grynspan commented Jun 13, 2024

Uh oh!

grynspan commented Jun 20, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

Uh oh!

Uh oh!

[SWT-0002] A stable JSON-based ABI for tools integration #479

[SWT-0002] A stable JSON-based ABI for tools integration #479

Uh oh!

Conversation

grynspan commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist:

Uh oh!

grynspan commented Jun 13, 2024

Uh oh!

plemarquand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

grynspan commented Jun 13, 2024

Uh oh!

grynspan commented Jun 20, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

grynspan commented Jul 10, 2024

Uh oh!

Uh oh!

Uh oh!

grynspan commented Jun 13, 2024 •

edited

Loading