Code coverage output not working with latest swift-testing and 6.0 toolchain

[xtremekforever]

Description

In Swift 5.10 we used the XCTest scaffold with swift-testing, then ran the tests with swift test --enable-code-coverage to get code coverage results, and this worked fine. However, when running the swift-testing tests directly with the latest Swift 6.0 toolchain, this code coverage output shows that no lines are covered.

Is this something that is planned on being implemented, or perhaps just not quite working yet in the new testing framework?

To confirm that this was not just the app I was developing (proprietary), I created a simple sample app that has a single function and tests it using the swift-testing framework. I will upload it so it can be used to see that the codecov report is showing no code coverage.

Expected behavior

Building the app with swift testing --enable-code-coverage generates the codecov output JSON with lines covered by tests.

Actual behavior

Got a codecov output JSON file showing that 0 lines are covered. For example:

{
    "branches": [],
    "expansions": [],
    "filename": "~/tmp/swift-testing-example/Sources/main.swift",
    "segments": [
        [
            8,
            53,
            0,
            true,
            true,
            false
        ],
        [
            9,
            8,
            0,
            true,
            true,
            false
        ],
        [
            9,
            26,
            0,
            true,
            false,
            false
        ],
        [
            9,
            27,
            0,
            true,
            true,
            false
        ],
        [
            11,
            6,
            0,
            true,
            false,
            false
        ],
        [
            11,
            12,
            0,
            true,
            true,
            false
        ],
        [
            13,
            6,
            0,
            true,
            false,
            false
        ],
        [
            14,
            2,
            0,
            false,
            false,
            false
        ],
        [
            16,
            1,
            0,
            true,
            true,
            false
        ],
        [
            16,
            50,
            0,
            false,
            false,
            false
        ],
        [
            17,
            1,
            0,
            true,
            true,
            false
        ],
        [
            17,
            14,
            0,
            false,
            false,
            false
        ]
    ],
    "summary": {
        "branches": {
            "count": 0,
            "covered": 0,
            "notcovered": 0,
            "percent": 0
        },
        "functions": {
            "count": 2,
            "covered": 0,
            "percent": 0
        },
        "instantiations": {
            "count": 2,
            "covered": 0,
            "percent": 0
        },
        "lines": {
            "count": 9,
            "covered": 0,
            "percent": 0
        },
        "regions": {
            "count": 6,
            "covered": 0,
            "notcovered": 6,
            "percent": 0
        }
    }
},

Steps to reproduce

1. Download and extract the code example: swift-testing-example.tar.gz
2. Using a recent Swift 6.0 toolchain (I used main-snapshot-2024-06-13), run swift test --enable-code-coverage.
3. After tests build and run, go to .build/debug/codecov and open the swift-testing-example.json file. It may be useful to prettify the JSON to make it more readable.
4. Observe that the entries for main.swift and the myTestFunction show 0 lines covered, wheras we would expect at least a few lines to be covered by the tests in the Tests/ directory.

swift-testing version/commit hash

0.10.0/69d59cfc76e5daf498ca61f5af409f594768eef9

Swift & OS version (output of swift --version ; uname -a)

Swift version 6.0-dev (LLVM 57177aa1b91540b, Swift 8be62863326595c)
Target: x86_64-unknown-linux-gnu
Linux server 6.1.0-21-amd64 swiftlang/swift-testing#1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

[stmontgomery]

Thank you for the report. We're aware of this issue but did not yet have an issue tracking it. Until it's fixed, you should be able to work around it by passing --disable-xctest, assuming you don't mind skipping any XCTests.

Transferring the issue to swift-package-manager repo, since the anticipated fix will be needed there.

[stmontgomery]

Tracked internally as rdar://130019309

[xtremekforever]

Thank you! I used the --disable-xctest flag and got the code coverage report that was expected! This works great for us since we do not use XCTest and we want to start using Swift 6.0 ahead of its release. Thanks again.

[stmontgomery]

Thank you! I used the --disable-xctest flag and got the code coverage report that was expected! This works great for us since we do not use XCTest and we want to start using Swift 6.0 ahead of its release. Thanks again.

Great! Glad the workaround was successful. We'll be keeping this issue open to track fixing this problem in SwiftPM. (Of course, you may also continue using that flag if you want, even after the fix lands.)

[grynspan]

This should be resolved with #7766.

[grynspan]

Resolved by #7789.