Update Smarter Testing docs #9795
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liamclarkedev
force-pushed
the
gordon/smarter-testing-updates
branch
3 times, most recently
from
19e6899 to
fdcd26e
Compare
jvincent42
force-pushed
the
gordon/smarter-testing-updates
branch
2 times, most recently
from
9715df1 to
35d471f
Compare
gordonsyme
force-pushed
the
gordon/smarter-testing-updates
branch
11 times, most recently
from
f051968 to
55fc45f
Compare
Rewrite the on-boarding docs now that we have a local CLI Rename adaptive-testing to test-impact-analysis Rename dynamic-batching to dynamic-test-splitting Co-authored-by: Liam Clarke <[email protected]> Co-authored-by: Jérémy Vincent <[email protected]>
gordonsyme
force-pushed
the
gordon/smarter-testing-updates
branch
from
55fc45f to
48b2160
Compare
jvincent42
approved these changes
Dec 2, 2025
|
|
||
| * Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then Smarter Testing may not be a good fit. | ||
| * Smarter Testing works best when testing a single deployable artifact. For example, a monorepo with integration tests that span multiple packages/services, especially when services run in separate container, makes coverage generation and consolidation difficult. | ||
| * Smarter Testing works best when testing a single deployable artifact. For example, a monorepo with integration tests that span multiple packages/services, especially when services run in separate containers, makes coverage generation and consolidation difficult. |
Contributor
There was a problem hiding this comment.
I've received feedback that this part is not very clear. And if meant all tests needed to be in a single job
Contributor
rosieyohannan
left a comment
rosieyohannan
left a comment
There was a problem hiding this comment.
OK I have left a bunch of suggestions and questions up to this point. I have not yet fully reviewed from section 4. Will do once we handle these changes
Comment on lines
+46
to
+48
| . Discovering tests. | ||
| . Running selected tests. | ||
| . Analysing test impact. |
Contributor
There was a problem hiding this comment.
Suggested change
| . Discovering tests. | |
| . Running selected tests. | |
| . Analysing test impact. | |
| * Discovering tests. | |
| * Running selected tests. | |
| * Analysing test impact. |
| Dynamic test splitting distributes your tests across parallel execution nodes. The system maintains a shared queue that each node pulls from to create a balanced workload. | ||
|
|
||
| When you configure parallelism in your job, Smarter Testing automatically: | ||
| When you configure parallelism in your job and enable Dynamic test splitting, Smarter Testing automatically: |
Contributor
There was a problem hiding this comment.
Suggested change
| When you configure parallelism in your job and enable Dynamic test splitting, Smarter Testing automatically: | |
| When you configure parallelism in your job and enable dynamic test splitting, Smarter Testing automatically: |
| This approach prevents slower nodes from extending the job runtime while other nodes have finished. All nodes continue to run tests until the entire test suite has been completed. | ||
|
|
||
| === Test impact analysis | ||
| Test impact analysis identifies which tests need to run based on the files that changed in your checked out code, stored as impact data for future test runs. The system works in two phases: |
Contributor
There was a problem hiding this comment.
Suggested change
| Test impact analysis identifies which tests need to run based on the files that changed in your checked out code, stored as impact data for future test runs. The system works in two phases: | |
| Test impact analysis identifies which tests need to run based on the files that changed in your checked out code. Analysis results are stored as impact data for future test runs. The system works in two phases: |
Contributor
There was a problem hiding this comment.
felt like these was something missing in this sentence from the last version!
| *Analysis phase*:: Builds a mapping between your tests and the code they exercise. Each test is run individually with code coverage enabled to determine which files it covers. Analysis runs on your default branch, but you can configure it to run on any branch with any trigger (webhook, API, or scheduled pipeline). | ||
|
|
||
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default, this runs on feature branches, but you can customize which branches use selection mode through your CircleCI configuration. | ||
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default test selection is applied on feature branches, and all tests are run on your default branch. You can customize this behavior in your CircleCI configuration. |
Contributor
There was a problem hiding this comment.
Suggested change
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default test selection is applied on feature branches, and all tests are run on your default branch. You can customize this behavior in your CircleCI configuration. | |
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default, test selection is applied on feature branches and all tests are run on your default branch. You can customize this behavior in your CircleCI configuration. |
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default, this runs on feature branches, but you can customize which branches use selection mode through your CircleCI configuration. | ||
| *Selection phase*:: Compares changed files against the test impact data and selects only tests that exercise modified code. By default test selection is applied on feature branches, and all tests are run on your default branch. You can customize this behavior in your CircleCI configuration. | ||
|
|
||
| Test selection works by comparing the current state of the repository against the most recent test impact analysis data. Any tests that cover modified files will be selected for running. Additionally the entire test suite will be selected if any of the `full-test-run-paths` files are modified, see the <<test-suite-configuration-options,test suite configuration options>> for details. |
Contributor
There was a problem hiding this comment.
Suggested change
| Test selection works by comparing the current state of the repository against the most recent test impact analysis data. Any tests that cover modified files will be selected for running. Additionally the entire test suite will be selected if any of the `full-test-run-paths` files are modified, see the <<test-suite-configuration-options,test suite configuration options>> for details. | |
| Test selection works by comparing the current state of the repository against the most recent test impact analysis data. Any tests that cover modified files are selected to run. Additionally, the entire test suite is selected if any of the `full-test-run-paths` files are modified. See the <<test-suite-configuration-options>> for details. |
Comment on lines
+368
to
370
| . Update the `.circleci/test-suites.yml` with the run command. | ||
| . In order to upload test results in your CI jobs, the location of the JUnit output file path also needs to be set in the test suite configuration. Set `outputs.junit` in `.circleci/test-suites.yml` with your preferred JUnit output file path. | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| . Update the `.circleci/test-suites.yml` with the run command. | |
| . In order to upload test results in your CI jobs, the location of the JUnit output file path also needs to be set in the test suite configuration. Set `outputs.junit` in `.circleci/test-suites.yml` with your preferred JUnit output file path. | |
| . Update the `.circleci/test-suites.yml` with the run command. | |
| . In order to upload test results in your CI jobs, the location of the JUnit output file path also needs to be set in the test suite configuration. Set `outputs.junit` in `.circleci/test-suites.yml` with your preferred JUnit output file path: | |
| + |
Contributor
There was a problem hiding this comment.
just to bring the following tabs into this numbered step
| ==== | ||
|
|
||
| === 2.2 Populate the analysis command | ||
| Run the test suite and confirm that the `run` command runs the test atoms you expect: |
Contributor
There was a problem hiding this comment.
Suggested change
| Run the test suite and confirm that the `run` command runs the test atoms you expect: | |
| . Run the test suite and confirm that the `run` command runs the test atoms you expect: | |
| + |
|
|
||
| We recommend following the steps in <<getting-started>> first before enabling the Smarter Testing feature to ensure the `discover` and `run` commands are set up correctly. | ||
|
|
||
| Steps: |
Contributor
There was a problem hiding this comment.
I feel a bit lost at this point. Are the following steps outlining what we are showing people how to do in the next couple of numbered steps 2.1 and 2.2? If so I can help make this clear
| . Test selection is driven from test impact analysis data. | ||
| . The analysis phase is correctly analysing test impact. | ||
|
|
||
| The next step is to run your test suite in CI. |
Contributor
There was a problem hiding this comment.
Suggested change
| The next step is to run your test suite in CI. | |
| The next step is to run your test suite in CI. This is covered in step 3. | |
| The following troubleshooting steps are provided to debug any issues you are finding up to this point. |
| . Update your `.circleci/config.yml` to call the `circleci run testsuite "ci tests"` command instead of your regular test command. | ||
| . Push the change to your VCS. | ||
|
|
||
| E.g. if your CircleCI test job was: |
Contributor
There was a problem hiding this comment.
Suggested change
| E.g. if your CircleCI test job was: | |
| For example, if your CircleCI test job was: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rewrite the on-boarding docs now that we have a local CLI