Skip to content

Add a new _TestDiscovery library/target. #981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Mar 5, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
6ae4139
Add a new _TestDiscovery library/target.
grynspan Feb 27, 2025
a35ffb7
Add link to GitHub issue asking for static layout validation
grynspan Feb 27, 2025
95fa659
Move HMODULE enumeration to new target
grynspan Feb 27, 2025
4bae45d
Try to build as a static library
grynspan Feb 27, 2025
70407fc
Make HMODULE.all package-visibility since we still need it in the mai…
grynspan Feb 27, 2025
de87958
Simplify var context
grynspan Feb 27, 2025
f44b6ae
package import
grynspan Feb 27, 2025
5f906ca
imageAddress type table
grynspan Feb 27, 2025
04c164a
Silence warning about package import on Windows
grynspan Feb 28, 2025
da214be
Copy more from CMake for _TestingInternals (?)
grynspan Feb 28, 2025
b7ec1a8
Emit module interface
grynspan Feb 28, 2025
1997bd1
Okay, we *do* need the package keyword here, the warning is just spur…
grynspan Feb 28, 2025
e0ef534
Silence package import warning for realsies
grynspan Feb 28, 2025
0de2121
Add lib prefix to static library (needed explicitly on Windows)
grynspan Feb 28, 2025
f7fae0d
Add CustomStringConvertible conformance to TestContentRecord
grynspan Mar 1, 2025
499bf11
Record address delta as hex
grynspan Mar 1, 2025
7451e42
Update some comments/messages
grynspan Mar 2, 2025
793608b
record -> recordAddress
grynspan Mar 2, 2025
2ede884
Update docs
grynspan Mar 2, 2025
32a51f1
Try to rationalize sendability
grynspan Mar 2, 2025
7ce0121
Doc tweaks
grynspan Mar 3, 2025
43437fa
Tweak imports of the new library to make them as private as possible
grynspan Mar 3, 2025
7366ec7
Incorporate feedback
grynspan Mar 3, 2025
adff9e9
Suppress DiscoverableAsTestContent protocol from .swiftinterface files
grynspan Mar 4, 2025
479149a
Remove @_spiOnly
grynspan Mar 4, 2025
5dc5d10
Lower ExitTest's DiscoverableAsTestContent conformance to internal
grynspan Mar 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 78 additions & 1 deletion Documentation/ABI/TestContent.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,9 @@ Third-party test content should set the `kind` field to a unique value only used
by that tool, or used by that tool in collaboration with other compatible tools.
At runtime, Swift Testing ignores test content records with unrecognized `kind`
values. To reserve a new unique `kind` value, open a [GitHub issue](https://github.com/swiftlang/swift-testing/issues/new/choose)
against Swift Testing.
against Swift Testing. The value you reserve does not need to be representable
as a [FourCC](https://en.wikipedia.org/wiki/FourCC) value, but it can be helpful
for debugging purposes.

The layout of third-party test content records must be compatible with that of
`TestContentRecord` as specified above. Third-party tools are ultimately
Expand All @@ -213,3 +215,78 @@ TODO: elaborate further, give examples
TODO: standardize a mechanism for third parties to produce `Test` instances
since we don't have a public initializer for the `Test` type.
-->

## Discovering previously-emitted test content

<!--
TODO: add more detail here about how to set up a package
-->

To add test content discovery support to your package, add a dependency on the
`_TestDiscovery` module in the `swift-testing` package (not the copy of Swift
Testing included with the Swift toolchain or Xcode), then import the module with
SPI enabled:

```swift
@_spi(Experimental) @_spi(ForToolsIntegrationOnly) import _TestDiscovery
```

> [!IMPORTANT]
> Don't add a dependency on the `swift-testing` package's `Testing` module. If
> you add a dependency on this module, it will cause you to build and link Swift
> Testing every time you build your package. You only need the `_TestDiscovery`
> module in order to discover your own test content types.

After importing `_TestDiscovery`, find the type in your module that should be
discoverable at runtime and add conformance to the `DiscoverableAsTestContent`
protocol:

```swift
extension FoodTruckDiagnostic: DiscoverableAsTestContent {
static var testContentKind: UInt32 { /* Your `kind` value here. */ }
}
```

This type does not need to be publicly visible. However, if the values produced
by your accessor functions are members of a public type, you may be able to
simplify your code by using the same type.

If you have defined a custom `context` type other than `UInt`, you can specify
it here by setting the associated `TestContentContext` type. If you have defined
a custom `hint` type for your accessor functions, you can set
`TestContentAccessorHint`:

```swift
extension FoodTruckDiagnostic: DiscoverableAsTestContent {
static var testContentKind: UInt32 { /* Your `kind` value here. */ }

typealias TestContentContext = UnsafePointer<FoodTruck.Name>
typealias TestContentAccessorHint = String
}
```

If you customize `TestContentContext`, be aware that the type you specify must
have the same stride and alignment as `UInt`.

When you are done configuring your type's protocol conformance, you can then
enumerate all test content records matching it as instances of
`TestContentRecord`.

You can use the `context` property to access the `context` field of the record
(as emitted into the test content section). The testing library will
automatically cast the value of the field to an instance of `TestContentContext`
for you.

If you find a record you wish to resolve to an instance of your conforming type,
call its `load()` function. `load()` calls the record's accessor function and,
if you have set a hint type, lets you pass an optional instance of that type:

```swift
for diagnosticRecord in FoodTruckDiagnostic.allTestContentRecords() {
if diagnosticRecord.context.pointee == .briansBranMuffins {
if let diagnostic = diagnosticRecord.load(withHint: "...") {
diagnostic.run()
}
}
}
```
34 changes: 28 additions & 6 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -25,22 +25,36 @@ let package = Package(
.visionOS(.v1),
],

products: [
{
products: {
var result = [Product]()

#if os(Windows)
result.append(
.library(
name: "Testing",
type: .dynamic, // needed so Windows exports ABI entry point symbols
targets: ["Testing"]
)
)
#else
result.append(
.library(
name: "Testing",
targets: ["Testing"]
)
)
#endif
}()
],

result.append(
.library(
name: "_TestDiscovery",
type: .static,
targets: ["_TestDiscovery"]
)
)

return result
}(),

dependencies: [
.package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.0-latest"),
Expand All @@ -50,6 +64,7 @@ let package = Package(
.target(
name: "Testing",
dependencies: [
"_TestDiscovery",
"_TestingInternals",
"TestingMacros",
],
Expand Down Expand Up @@ -101,13 +116,20 @@ let package = Package(
]
),

// "Support" targets: These contain C family code and are used exclusively
// by other targets above, not directly included in product libraries.
// "Support" targets: These targets are not meant to be used directly by
// test authors.
.target(
name: "_TestingInternals",
exclude: ["CMakeLists.txt"],
cxxSettings: .packageSettings
),
.target(
name: "_TestDiscovery",
dependencies: ["_TestingInternals",],
exclude: ["CMakeLists.txt"],
cxxSettings: .packageSettings,
swiftSettings: .packageSettings
),

// Cross-import overlays (not supported by Swift Package Manager)
.target(
Expand Down
1 change: 1 addition & 0 deletions Sources/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ endif()

include(AvailabilityDefinitions)
include(CompilerSettings)
add_subdirectory(_TestDiscovery)
add_subdirectory(_TestingInternals)
add_subdirectory(Overlays)
add_subdirectory(Testing)
7 changes: 4 additions & 3 deletions Sources/Testing/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,7 @@ add_library(Testing
Support/Locked.swift
Support/Locked+Platform.swift
Support/Versions.swift
Discovery.swift
Discovery+Platform.swift
Discovery+Macro.swift
Test.ID.Selection.swift
Test.ID.swift
Test.swift
Expand All @@ -106,6 +105,7 @@ add_library(Testing
Traits/TimeLimitTrait.swift
Traits/Trait.swift)
target_link_libraries(Testing PRIVATE
_TestDiscovery
_TestingInternals)
if(NOT APPLE)
if(NOT CMAKE_SYSTEM_NAME STREQUAL WASI)
Expand All @@ -120,8 +120,9 @@ if(NOT APPLE)
endif()
if(NOT BUILD_SHARED_LIBS)
# When building a static library, tell clients to autolink the internal
# library.
# libraries.
target_compile_options(Testing PRIVATE
"SHELL:-Xfrontend -public-autolink-library -Xfrontend _TestDiscovery"
"SHELL:-Xfrontend -public-autolink-library -Xfrontend _TestingInternals")
endif()
add_dependencies(Testing
Expand Down
48 changes: 48 additions & 0 deletions Sources/Testing/Discovery+Macro.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
//
// This source file is part of the Swift.org open source project
//
// Copyright (c) 2023–2025 Apple Inc. and the Swift project authors
// Licensed under Apache License v2.0 with Runtime Library Exception
//
// See https://swift.org/LICENSE.txt for license information
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
//

@_spi(Experimental) @_spi(ForToolsIntegrationOnly) internal import _TestDiscovery

/// A shadow declaration of `_TestDiscovery.DiscoverableAsTestContent` that
/// allows us to add public conformances to it without causing the
/// `_TestDiscovery` module to appear in `Testing.private.swiftinterface`.
///
/// This protocol is not part of the public interface of the testing library.
protocol DiscoverableAsTestContent: _TestDiscovery.DiscoverableAsTestContent, ~Copyable {}

/// The type of the accessor function used to access a test content record.
///
/// The signature of this function type must match that of the corresponding
/// type in the `_TestDiscovery` module. For more information, see
/// `ABI/TestContent.md`.
///
/// - Warning: This type is used to implement the `@Test` macro. Do not use it
/// directly.
public typealias __TestContentRecordAccessor = @convention(c) (
_ outValue: UnsafeMutableRawPointer,
_ type: UnsafeRawPointer,
_ hint: UnsafeRawPointer?
) -> CBool

/// The content of a test content record.
///
/// The layout of this type must match that of the corresponding type
/// in the `_TestDiscovery` module. For more information, see
/// `ABI/TestContent.md`.
///
/// - Warning: This type is used to implement the `@Test` macro. Do not use it
/// directly.
public typealias __TestContentRecord = (
kind: UInt32,
reserved1: UInt32,
accessor: __TestContentRecordAccessor?,
context: UInt,
reserved2: UInt
)
Loading