-
+
Detect and manage changes
Compare reports in detail to review only _new_ accessibility issues, use
diff --git a/docs/accessibility/guides/detect-changes.mdx b/docs/accessibility/guides/detect-changes.mdx
index b7680ed1d3..3015a7e085 100644
--- a/docs/accessibility/guides/detect-changes.mdx
+++ b/docs/accessibility/guides/detect-changes.mdx
@@ -11,7 +11,7 @@ sidebar_position: 70
Cypress Accessibility supports three main paths to observing and acting on changes in the accessibility report for your projects:
-- Run comparisons with [Branch Review](/accessibility/core-concepts/comparing-reports)
+- Run comparisons with [Branch Review](/accessibility/core-concepts/compare-reports)
- The [Results API](/accessibility/results-api) to fail builds or trigger alerts when results don't meet your standards
- [Analytics](/cloud/features/analytics/enterprise-reporting#Cypress-Accessibility) for high-level trend-spotting and analysis
@@ -33,7 +33,7 @@ Use Branch review when reviewing a pull request made by another developer or bef
Since this reduces the amount of violations to review to just what was affected by the PR code changes, you won't spend any time looking for what's relevant -- and if there is no difference, you'll know your PR does not affect the accessibility score and is safe to merge.
-See the main [Accessibility Branch Review docs](/accessibility/core-concepts/comparing-reports) for a video demo of this workflow.
+See the main [Accessibility Branch Review docs](/accessibility/core-concepts/compare-reports) for a video demo of this workflow.
#### Compare reports during local development (without waiting for CI)
diff --git a/docs/app/get-started/why-cypress.mdx b/docs/app/get-started/why-cypress.mdx
index a5561e91ac..b8e7c9d4d8 100644
--- a/docs/app/get-started/why-cypress.mdx
+++ b/docs/app/get-started/why-cypress.mdx
@@ -108,12 +108,14 @@ Below are listed some of the key features of each product.
- **Visualize Coverage:** [UI Coverage](/ui-coverage/get-started/introduction) provides a visual overview of test coverage across every page and component of your app, offering clear insights into uncovered areas that everyone can understand.
- **Results API:** Use the UI Coverage [Results API](/ui-coverage/results-api) to programmatically access test coverage data and integrate it into your existing workflows.
+- **Branch Review:** Compare runs to see newly introduced elements in your application or unexpected reductions in test coverage.
### Cypress Accessibility
- **Accessibility Checks:** Maximize the value of your existing Cypress tests by instantly adding thousands of [accessibility checks](/accessibility/get-started/introduction) with no setup, code changes, or performance penalty.
- **Run-level reports:** Get a detailed report of accessibility issues found in your test runs with [Run-level reports](/accessibility/core-concepts/run-level-reports).
- **Results API:** Use the Cypress Accessibility's [Results API](/accessibility/results-api) to programmatically access Accessibility results in a CI environment.
+- **Branch Review:** Compare any report against a baseline to review only the new violations, without any noise from existing issues.
## Solutions
diff --git a/docs/app/guides/accessibility-testing.mdx b/docs/app/guides/accessibility-testing.mdx
index d65a17557f..26bdf90b23 100644
--- a/docs/app/guides/accessibility-testing.mdx
+++ b/docs/app/guides/accessibility-testing.mdx
@@ -92,6 +92,10 @@ To learn more, you can read our [dedicated docs](/accessibility/get-started/intr
While automation like Axe Core® can detect missing attributes and other aspects of code quality that impact the experience of people with disabilities using the web, this kind of automation doesn't know anything about your specific application and the expectations you have for your users. That's where including accessibility in your specs comes in.
+### Keyboard navigation
+
+To test in-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events. This can also be used to test custom behavior like autocomplete.
+
### Asserting alt text of images
To assert that the correct alternative text is present on your logo image, an explicit test can be written:
@@ -169,10 +173,6 @@ This means that, without some assertions about either the `button` element itsel
Developers who are unfamiliar with accessibility may assume that if a Testing Library `ByRole` locator can be made to pass before and after a code change, there has been no functional or accessibility-related change in the underlying element. As we've seen, this is not actually the case, because of the extra behavior browsers only implement for native HTML elements. For more about this difference and why semantic HTML elements are preferred, see the [first rule of Accessible Rich Internet Applications (ARIA)](https://www.w3.org/TR/using-aria/#rule1).
-### Keyboard navigation
-
-To test intra-page navigation with the keyboard, you can use the [`cy.press()`](/api/commands/press) to dispatch native tab events.
-
## Where to test accessibility
So what should you do in your test automation to help confirm the UI is accessible? First of all, for known critical areas like forms or checkout flows, ensure that the accessibility behavior is tested explicitly in at least one place. The means verifying that form fields and buttons have the correct labels and use the expected HTML elements, and other aspects of the DOM that communicate necessary information.
diff --git a/docs/app/references/trade-offs.mdx b/docs/app/references/trade-offs.mdx
index 061bc77055..cf0e2a0d52 100644
--- a/docs/app/references/trade-offs.mdx
+++ b/docs/app/references/trade-offs.mdx
@@ -37,7 +37,6 @@ We want to highlight some _temporary_ restrictions that Cypress hopes to
eventually address.
- [Workarounds for the lack of a `cy.hover()` command.](/api/commands/hover)
-- [`cy.tab()` command.](https://github.com/cypress-io/cypress/issues/299)
- [There is not any native or mobile events support.](https://github.com/cypress-io/cypress/issues/311#issuecomment-339824191)
- [iframe support is somewhat limited, but does work.](https://github.com/cypress-io/cypress/issues/136)
diff --git a/docs/cloud/features/branch-review.mdx b/docs/cloud/features/branch-review.mdx
index 8b6df8cd48..413517bf2b 100644
--- a/docs/cloud/features/branch-review.mdx
+++ b/docs/cloud/features/branch-review.mdx
@@ -1,6 +1,6 @@
---
title: Branch Review
-description: Branch Review in Cypress Cloud surfaces the impact of pull requests on your test suite. Compare test results between branches and learn best practices for grouping test runs.
+description: Branch Review in Cypress Cloud surfaces the impact of Pull Requests on your test suite. Compare test results between branches and learn best practices for grouping test runs.
sidebar_position: 50
---
@@ -12,114 +12,133 @@ sidebar_position: 50
##### What you'll learn
-- How Branch Review surfaces the impact of pull requests on your test suite
- How to compare test results between branches in Cypress Cloud
+- How Branch Review surfaces the impact of Pull Requests on your test suite
+- How to compare **any** two runs, if not using a Pull Request workflow
- Best practices for grouping test runs
:::
-Cypress Branch Review is designed to elevate your pull request review workflow. It allows you to quickly identify the impact a pull request might have on your test suite in a single view. Compare which tests are failing, flaky, pending, added, or modified between the source and base branches and prevent the merging of low-quality code.
+Cypress Branch Review allows you to quickly identify the different impacts a Pull Request might have on your test suite in a single view.
+Compare which tests are failing, flaky, pending, added, or modified between the source and base branches. If enabled on your account,
+UI Coverage or Cypress Accessibility changes between runs will also appear here.
-A common scenario throughout the software development lifecycle (SDLC) is an engineer's **feature** branch that will be merged into the repo's **develop** or **main** branch.
+Branch Review is useful even if you do not have a Pull Request workflow, because it allows you to compare any two runs of your choice.
+These might might represent different builds of the application, changes in test code, or results from nightly runs. This page will use
+the example of a Pull Request made using the GitHub integration, but this integration is not required in order to use Branch Review.
+
+A common scenario throughout the software development lifecycle (SDLC) is an engineer's **feature** branch that will be merged into the
+repo's **develop** or **main** branch. Here's how this scenario appears in Branch Review:
-Previously, pinpointing changes in your test suite's results required a manual side-by-side comparison between your newly-introduced branch's test runs and your base branch's test runs. This sub-optimal workflow often fails to answer the fundamental questions, _what changed_ and _why?_ You might be left wondering if the same tests are flaky between branches, when new failures were introduced, or if you added sufficient test coverage to your new branch.
-
-:::info
-
-Cypress Branch Review is currently only available for [GitHub integrated projects](/cloud/integrations/source-control/github). We are working on adding support for GitLab and Bitbucket soon.
-:::
+Without Branch Review, pinpointing changes in your test suite's results requires a manual side-by-side comparison between your newly-introduced branch's test runs and your base branch's test runs. This sub-optimal workflow often fails to answer the fundamental questions, _what changed_ and _why?_ You might be left wondering if the same tests are flaky between branches, when new failures were introduced, or if you added sufficient test coverage to your new branch.
## Getting Started
-Branch Review works by leveraging the Cypress Cloud GitHub integration to query the GitHub API for branches with pull requests. Your project will first need to be connected to a GitHub repository. If you haven't already, follow the steps in [Install the Cypress GitHub app](/cloud/integrations/source-control/github#Install-the-Cypress-GitHub-app) to connect your project to GitHub.
+The key to the Branch Review area is a comparison between a **Base** run and a **Changed** run.
+The Changed run is the main subject of the comparison, often associated with an incoming Pull Request. The Base run is the starting point for comparison.
### Accessing Branch Review
-To access Branch Review, navigate to the **Branches** tab in the left sidebar and select the branch you want to review. In order to see a comparison, the branch you select must have a pull request.
+There are a number of ways to get to Branch Review depending on where you are looking at your Cypress results. In all cases, once you've picked Changed run, you can adjust the automatically-selected Base run to any branch or run in the project to dial in the comparison you are most interested in.
+
+### From the project list
+
+The project card shows three active branches for the project. You can click through to any of these to enter Branch Review focused on that branch.
-If you do not see PR # tags associated with a branch, you may need to [pass PR numbers](/app/continuous-integration/overview#CI-Build-Information) to runs as environment variables. This helps ensure Cypress Cloud can map PR data correctly.
-
-#### Latest Runs
+### From the main navigation
-If the selected branch has a PR associated and a PR # passed via CI, there is an additional callout on the top of the overview tab of the **Latest runs** page which directs to the Branch Review.
+When a project is open, select "Branches" in the main navigation to see a full, filterable list of available branches and choose one to set as the source of the Changed run.
-## Branch Details
+### From the run overview page
+
+Click the branch name associated with the run. This will take you to Branch Review with that run pre-selected as the Changed run. This is a great way to investigate a specific run with an unexpected outcome in the tests.
+
+
-### View Options
+### From a Pull Request
-"Overview" is the default view of a branch for all projects.
+Click the "View all changes introduced in this branch" link at the bottom of the Test Results table. You will enter branch review with the current PR's branch and latest run pre-selected as the Changed run.
-:::note
+## Integrating with Pull Requests
-**Branch Overview**
+Pull Requests in Branch Review work by leveraging the Cypress Cloud GitHub integration to query the GitHub API for branches with Pull Requests. Your project will first need to be connected to a GitHub repository. If you haven't already, follow the steps in [Install the Cypress GitHub app](/cloud/integrations/source-control/github#Install-the-Cypress-GitHub-app) to connect your project to GitHub.
-1. Callouts for the latest available Branch Review or a reminder to create a pull request in order to compare test run results.
+When Pull Requests are connected, you will be able to select runs by the associated PR:
-2. Click "Review changes" for the full Branch Review.
+
-3. Click any test status to reveal those changes.
+:::info
-4. Latest runs of the current branch for contextual guidance on the state of its test runs.
+**Note:** If you do not see PR # tags associated with a branch, you may need to [pass PR numbers](/app/continuous-integration/overview#CI-Build-Information) to runs as environment variables. This helps ensure Cypress Cloud can map PR data correctly.
:::
-"Review" provides the detailed Branch Review insights for the branch's available pull requests, with the latest pull request selected by default.
+#### Latest Runs
+
+If the selected branch has a PR associated and a PR # passed via CI, there is an additional callout on the top of the overview tab of the **Latest runs** page which directs to the Branch Review.
+
+
-"Runs" provides a list of all runs that are attributed to the current branch.
+## Branch Details
### Review Header
The header includes the Git commit message, PR selector, PR status, and base and feature branch labels with the Cypress Cloud test run ID number (#).
-If there are multiple pull requests open for the same branch, you can select the pull request you want to review from the dropdown to the right of the commit message.
+If there are multiple Pull Requests open for the same branch, you can select the Pull Request you want to review from the dropdown to the right of the commit message.
Clicking the branch label will link you directly to the appropriate [run overview](/cloud/features/recorded-runs#Overview-tab) and hovering on these elements exposes additional run meta data.
### Review Test Status
-When on the review screen, you will see the Failures, Flaky, Pending, Added, and Modified tabs. Each tab will show you the specs that fall into that category. You can click on a test to view the test details.
-
:::info
-If a run is missing on either the merge base or feature commit of the PR, a banner is displayed specifying which commit is missing the run with a link to it on GitHub.
+**Note:** Recording exactly one run for every commit is the best way to ensure complete and accurate comparisons. See our [best practices](/cloud/features/branch-review#Best-Practices) on grouping multiple `cypress run` calls under one run.
:::
-:::info
-
-Recording exactly one run for every commit as the best/only way to ensure complete and accurate comparisons. See our [best practices](/cloud/features/branch-review#Best-Practices) on grouping multiple `cypress run` calls under one run.
-
-:::
+When on the review screen, you will see the Failures, Flaky, Pending, Added, and Modified tabs. Each tab will show you the specs that fall into that category. You can click on a test to view the test details.
@@ -127,9 +146,11 @@ Recording exactly one run for every commit as the best/only way to ensure comple
**Review Test Status**
-1. Review your PR information
-2. Get a glance of changes in your branch across key statuses
-3. See what has changed in a specific status, for example:
+1. Details of which branches and runs are compared. If a Pull Request is detected, Pull Request details will be here.
+1. Links to key differences in test information like new failures and flaky tests. Note that all numbers and scores refer to the Changed run.
+1. [UI Coverage](https://on.cypress.io/ui-coverage-trial?utm_medium=cloud-br&utm_source=docs.cypress.io&utm_term=&utm_content=UI+Coverage) changes.
+1. [Cypress Accessibility](http://on.cypress.io/accessibility-trial?utm_medium=cloud-br&utm_source=docs.cypress.io&utm_term=&utm_content=Cypress+Accessibility) changes.
+1. See what has changed in the specific status you have opened, for example:
- 
total count, for example, _3 new and 1 existing_
-4. The test state (failed, flaky, pending) is also indicated at the spec level
+1. The test state (failed, flaky, pending) is also indicated at the spec level
- _new_ = the state was not previously captured, but now is captured
(newly pending tests can imply an `it.skip()` was not removed)
- _existing_ = the state was previously captured and now is still captured
- _resolved_ = the state was previously captured but now is no longer captured
@@ -177,19 +198,9 @@ Navigating into the test detail view reveals a side-by-side comparison of the te
:::
-## GitHub Pull Request Comments
-
-The Cypress Cloud GitHub Integration offers detailed run information via [PR comments](/cloud/integrations/source-control/github#Pull-request-comments). Follow the "View all changes introduced in this branch" link to load the Branch Review between the feature and base branches.
-
-
-
## Troubleshooting
-### Note on Available Data
+### Note on Available Data for Pull Requests
Branch Review is a powerful tool to compare two branches with recorded runs to Cypress Cloud. There are factors that can impact what is available to review between a feature and base branch. For example, whether or not a branch at a particular commit has a run recorded to Cypress Cloud will affect what is displayed.
@@ -202,6 +213,8 @@ The following captures this scenario across both branches:
| no run | has run | Non-comparison data with found feature run |
| no run | no run | Non-comparison data using last run on feature branch |
+If the required runs for comparison are not available, you can use the manual selection dropdowns to compare two suitable runs, if they exist in Cypress Cloud.
+
## Best Practices
### Grouping Test Runs
@@ -210,6 +223,10 @@ Cypress Cloud allows for [grouping recorded tests](/cloud/features/smart-orchest
Recording multiple test runs per commit without grouping will cause issues, as Branch Review relies on the _latest_ run. Therefore, recording one run per commit and utilizing grouping is essential to improving effectiveness and ensure an accurate comparison between branches.
+### Missing Runs in Pull Request Comparisons
+
+If a run is missing on either the merge base or feature commit of the PR, a banner is displayed specifying which commit is missing the run with a link to it on GitHub.
+
## See Also
-- Read about [multi-repo implementation](/cloud/account-management/projects#Multi-repo-Implementation) best practices in Cypress Cloud.
+- Read about [multi-repo implementation](/cloud/account-management/projects#Multi-repo-Implementation) best practices in Cypress Cloud. Even if your test code and application code are not colocated, you maybe be able to connect your test runs to specific commits and pull requests in your application.
diff --git a/docs/cloud/get-started/introduction.mdx b/docs/cloud/get-started/introduction.mdx
index 3d51c6c61e..ffcea8393d 100644
--- a/docs/cloud/get-started/introduction.mdx
+++ b/docs/cloud/get-started/introduction.mdx
@@ -13,11 +13,13 @@ Cypress Cloud unlocks the full potential of Cypress test automation tools in you
+
### Flexible enterprise configuration and single sign on
diff --git a/docs/ui-coverage/core-concepts/compare-reports.mdx b/docs/ui-coverage/core-concepts/compare-reports.mdx
new file mode 100644
index 0000000000..723003109b
--- /dev/null
+++ b/docs/ui-coverage/core-concepts/compare-reports.mdx
@@ -0,0 +1,97 @@
+---
+title: 'Comparing UI Coverage Reports | Cypress Documentation'
+description: 'Review where coverage has changed in the application between two runs.'
+sidebar_position: 40
+sidebar_label: Compare reports
+---
+
+
+
+# Compare reports
+
+UI Coverage reports from different runs can be compared in the [Branch Review](/cloud/features/branch-review#Getting-Started) area of Cypress Cloud. This allows you to understand the exact change in coverage between any two points in time, revealing the impact of application changes or test code updates on your overall coverage area.
+
+
+
+## Use cases
+
+Comparing the results from different runs is useful in multiple scenarios.
+
+**Key use cases:**
+
+- **Pre-merge checks**: Know if any net-new untested elements are introduced by UI code changes.
+- **Monitoring changes**: Compare nightly monitoring runs and track down changes in coverage caused by unexpected changes in the application.
+- **Reviewing AI-generated code changes**: The increased use of AI to generate and/or review both tests and application code increases the risk of reducing coverage, or adding redundant coverage.
+- **Tracing the introduction of issues**: With dropdowns for each run, it's easy to rapidly compare different A and B runs to find the exact commit that introduced a problem.
+- **Demonstrating the resolution of issues**: Confirm the effect of your improvements, and share the overview with your team to more quickly review changes.
+
+## Content of the report
+
+The Branch Review report is organized into three sections:
+
+### Untested links
+
+
+
+Untested links represent unopened doors in your application - pages that a user can reach through your UI, but that are not tested with Cypress. This report shows any new untested pages that appear in the Changed run that were not present in the Base run.
+
+### Untested elements
+
+
+
+Untested interactive elements can increase for two potential reasons:
+
+1. In the Changed run, the tests are no longer interacting with an element that was interacted with in the Base run. The element is still present, but the **test code** has changed.
+2. In the Changed run, new elements exist that were not present in the Base run, and they are not tested in the Changed run. This happens when the **application code** has changed but tests have not kept up.
+
+Both changes are worth reviewing, and while the causes are different, the way to increase coverage is the same. The elements can be tested with Cypress to contribute towards your coverage score, or they can be ignored with configuration if it is not important to test the elements.
+
+### Added links, elements, and views
+
+
+
+This is a summary of everything that appears to be new in the Changed run, and includes both tested and untested items. It's a useful way to compare two builds of your application and how the changes in the application itself relate to your overall coverage metrics.
+
+### Detail View
+
+
+
+Clicking on any untested link or element will take you into the Detail View, where you can see all the examples of the specific elements that have changed between the two runs. Elements that have the same status on both runs are removed here, so you can focus only on the changes.
+
+## How to compare runs
+
+The first step is to get to the [Branch Review](/cloud/features/branch-review.mdx) area of Cypress Cloud, which will let you compare one branch against another - or different runs on the same branch, if needed.
+You can access this area by clicking the branch name associated with a run, or in several other ways. [Learn more about how to compare runs](/cloud/features/branch-review.mdx).
+
+## FAQ
+
+### How do I ensure a good comparison?
+
+The best subjects to compare are passing runs that ran similar tests on the same set of content. This means that each run visited roughly the same pages and completed the same kinds of workflows. In this situation, any diff in the results is likely the result of changes present in the newer run. This is usually what happens out-of-the box when comparing a pull-request branch with your main branch.
+
+That said, it is still possible and valid to compare runs from different points in time with different sets of test results, as long as you bear in mind all the potential sources of difference between the two runs, which you can evaluate for yourself as you explore the results.
+
+In order to see unified changes for your entire test suite, you need to group all the tests together under a single Cypress run, for each report. Learn more about this in the [Branch Review Best Practices documentation](/cloud/features/branch-review#Best-Practices).
+
+### What is the purpose of the Beta label?
+
+This indicates the feature is ready for use and actively seeking feedback based on real usage of the current implementation. We have a few known issues to work through on our side before we consider this fully production-ready and remove the beta label. These issues only affect a subset of projects -- in most cases everything is working as intended. If you see anything unexpected, please use the feedback button and let us know.
+
+### Why do I see some views (pages or components) changing from run-to-run?
+
+URLs with dynamic slugs in them can appear as "new" pages in some situations. This behavior can be adjusted with [View configuration](/accessibility/configuration/views) to make sure page names will match across runs by wildcarding parts of the URL as needed.
diff --git a/docs/ui-coverage/get-started/introduction.mdx b/docs/ui-coverage/get-started/introduction.mdx
index 8c03460a97..e42be9e068 100644
--- a/docs/ui-coverage/get-started/introduction.mdx
+++ b/docs/ui-coverage/get-started/introduction.mdx
@@ -22,7 +22,7 @@ Easily track, monitor, and visualize the test coverage of your UI to prevent reg
label="See a demo"
icon="action-play-small"
className="mr-1"
- href="https://go.cypress.io/hubfs/App%20Quality%20Products/UI%20Coverage/ui_coverage__overview_and_demo%20(1080p).mp4"
+ href="https://on.cypress.io/ui-coverage-demo-video"
/>