feat(testing): add playwright setup #5860

rubencarvalho · 2025-11-06T15:36:03Z

Description

Trying to get this #5835 in - now rebased!

SWC-1263

Proof of concept implementing automated accessibility testing using Playwright with two complementary approaches:

1. ARIA Snapshot Testing

Captures and validates accessibility tree structure
Detects unintended changes to component semantics
Human-readable YAML format serves as living documentation

2. aXe-core WCAG Validation

Automated WCAG 2.0/2.1 Level A/AA compliance checking
Validates color contrast, ARIA attributes, keyboard accessibility
Focused on standards compliance (excludes best-practice rules)

Test Coverage

25 tests passing across both generations (~30s runtime):

1st Generation (13 tests):

Badge: 6 tests (3 ARIA snapshots, 3 aXe validations)
Status Light: 7 tests (3 ARIA snapshots, 4 aXe validations)

2nd Generation (12 tests):

Badge: 4 tests (2 ARIA snapshots, 2 aXe validations)
Status Light: 8 tests (3 ARIA snapshots, 5 aXe validations)

Key Features

Dual Storybook Support: Playwright projects handle both generations (1st gen: port 8080, 2nd gen: port 6006)
Deterministic Testing: No arbitrary timeouts (waits for custom element definition, visibility, upgrade)
Collocated Tests: Tests live with components in test/*.a11y.spec.ts
Shared Helpers: Generation-agnostic utilities in 1st-gen/test/a11y-helpers.ts work for both gens
Auto-start Storybooks: Both instances launch automatically when running tests
Consolidated Documentation: Single source of truth at ACCESSIBILITY_TESTING.md

Usage

yarn test:a11y          # Run all tests (both generations)
yarn test:a11y:1st      # Only 1st gen tests
yarn test:a11y:2nd      # Only 2nd gen tests
yarn test:a11y:ui       # Interactive Playwright UI

Files Added/Modified

Test Files:

1st-gen/packages/badge/test/badge.a11y.spec.ts
1st-gen/packages/status-light/test/status-light.a11y.spec.ts
2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts
2nd-gen/packages/swc/components/status-light/test/status-light.a11y.spec.ts

Shared Utilities:

1st-gen/test/a11y-helpers.ts - Deterministic helpers for both generations

Configuration:

1st-gen/playwright.config.ts - Dual-Storybook Playwright setup with projects
1st-gen/package.json - Test scripts (test:a11y, test:a11y:1st, test:a11y:2nd, test:a11y:ui)
package.json - Root-level convenience commands

Documentation:

ACCESSIBILITY_TESTING.md - Comprehensive guide covering quick start, how-to, and best practices

ARIA Snapshots (auto-generated baselines):

1st-gen/packages/badge/test/badge.a11y.spec.ts-snapshots/*.aria.yml
1st-gen/packages/status-light/test/status-light.a11y.spec.ts-snapshots/*.aria.yml
2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts-snapshots/*.aria.yml
2nd-gen/packages/swc/components/status-light/test/status-light.a11y.spec.ts-snapshots/*.aria.yml

Motivation and Context

Problem: Manual accessibility testing is time-consuming and inconsistent. Regressions can slip through unnoticed.

Solution: Automated testing catches accessibility issues early in development and serves as executable documentation of expected behavior.

Approach: This POC demonstrates the pattern on Badge and Status Light components across both 1st and 2nd generation. If approved, the pattern can be applied to all components.

Technical Decisions

1. WCAG-only scanning

Excludes best-practice rules (e.g., "page must have h1")
Focuses on WCAG 2.0/2.1 Level A/AA compliance
Reasoning: We test isolated components in Storybook, not full pages

2. Deterministic waits

Custom gotoStory() helper waits for specific conditions
No arbitrary timeouts or waitForLoadState('networkidle')
Faster, more reliable than waiting for network idle

3. Playwright projects for dual-generation support

Separate configurations for 1st gen (port 8080) and 2nd gen (port 6006)
Allows running tests independently or together
Auto-starts both Storybook instances as needed

4. Collocated tests

Tests live in each component's test/ directory
Follows existing test organization patterns
Easier discovery and maintenance

5. Shared, generation-agnostic helpers

Single helper library at 1st-gen/test/a11y-helpers.ts
Works for both 1st gen (sp-*) and 2nd gen (swc-*) components
No code duplication

Related Issue(s)

This is a proof-of-concept RFC for team discussion
No existing issue (greenfield implementation)

Author's Checklist

I have read the CONTRIBUTING and PULL_REQUESTS documents
I have reviewed the Accessibility Practices for this feature
I have added automated tests to cover my changes
I have included updated documentation

Reviewer's Checklist

Questions for Discussion

Approach: Do ARIA snapshots + aXe validation provide the right balance?
Coverage: Should we add these tests to all components or start with high-priority ones?
CI Integration: How should these run in CI? (All PRs? Scheduled? On-demand?)
Maintenance: Who owns keeping snapshots updated when designs change?
Snapshot Management: ARIA snapshots are committed - is this the right approach?

Manual Testing

Verify tests run successfully:

Clean environment: pkill -f "storybook"
Run all tests: cd 1st-gen && yarn test:a11y
Expected: 25 passing tests in ~30 seconds
Verify both Storybooks auto-start (ports 8080 and 6006)

Review test outputs:

Open HTML report: yarn test:a11y:report
Review ARIA snapshots in **/test/*-snapshots/ directories
Verify human-readable YAML format

Try individual commands:

yarn test:a11y:1st - Should run 14 1st gen tests
yarn test:a11y:2nd - Should run 11 2nd gen tests
yarn test:a11y:ui - Should open Playwright UI
yarn test:a11y badge --update-snapshots - Update baselines

Review documentation:

Read ACCESSIBILITY_TESTING.md for comprehensive guide
Follow "Adding tests to a component" section
Verify examples are clear and copy-paste ready

Note: This is a POC for discussion. Not intended for immediate merge - seeking feedback on approach and implementation before broader rollout.

changeset-bot · 2025-11-06T15:36:09Z

⚠️ No Changeset found

Latest commit: cf00510

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

rubencarvalho · 2025-11-06T15:48:45Z

1st-gen/packages/badge/test/badge.a11y.spec.ts

+
+import { expect, test } from '@playwright/test';
+import AxeBuilder from '@axe-core/playwright';
+import { gotoStory } from '../../../../2nd-gen/test/a11y-helpers.js';


I think we still don’t have a definitive place for cases like this, but this kind of inter-workspace import isn’t ideal (it could cause issues with CI or other configurations). For this specific case, though, gotoStory seems small enough that we could just include it in each generation?
Ideally, though, we would conceptually land on something like @spectrum-web-components/core/toolkit

Totally makes sense! I've addressed that here 580727a.

2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts

rubencarvalho · 2025-11-06T15:49:51Z

2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts

+import AxeBuilder from '@axe-core/playwright';
+import { expect, test } from '@playwright/test';
+
+import { gotoStory } from '../../../../../test/a11y-helpers.js';


Given that this is @adobe/swc (the actual components implementation) related-only, it feels like this helper could be moved 2nd-gen/packages/swc/utils

rubencarvalho · 2025-11-06T15:52:13Z

playwright.a11y.config.ts

nit: I don't have the full context, but couldn't this be on the root level? From what I've seen, it's only getting called from root?

It can! I moved it there and updated the commands in package.json. ✨

rubencarvalho · 2025-11-06T15:57:19Z

I didn’t re-add the tsconfig-all excludes because we do want TypeScript to run on our tests. We rely on it to verify certain functionality (e.g., tests will fail if unexpected usage occurs).

github-actions · 2025-11-06T17:01:02Z

📚 Branch Preview

🔍 Visual Regression Test Results

When a visual regression test fails (or has previously failed while working on this branch), its results can be found in the following URLs:

Deployed to Azure Blob Storage: pr-5860

If the changes are expected, update the current_golden_images_cache hash in the circleci config to accept the new images. Instructions are included in that file.
If the changes are unexpected, you can investigate the cause of the differences and update the code accordingly.

package.json

rubencarvalho added the Status: WIP PR is a work in progress or draft label Nov 6, 2025

rubencarvalho force-pushed the ruben/playwright-a11y-testing branch from 0ecadf3 to cf5ba2d Compare November 6, 2025 15:37

rubencarvalho commented Nov 6, 2025

View reviewed changes

2nd-gen/packages/swc/components/badge/test/badge.a11y.spec.ts Outdated Show resolved Hide resolved

rubencarvalho commented Nov 6, 2025

View reviewed changes

feat(testing): add playwright setup

1b9ea79

rubencarvalho force-pushed the ruben/playwright-a11y-testing branch from 05339c6 to 1b9ea79 Compare November 6, 2025 16:18

rubencarvalho and others added 4 commits November 6, 2025 17:22

chore: update yarn.lock

a8b305e

fix(a11y-testing-spike): remove cross workspace import

580727a

fix(a11y-testing-spike): clean up helper paths

3416bea

fix(a11y-testing-spike): move playwright a11y config to root

a1ba798

cdransf approved these changes Nov 6, 2025

View reviewed changes

cdransf changed the title ~~[WIP] feat(testing): add playwright setup~~ feat(testing): add playwright setup Nov 6, 2025

cdransf marked this pull request as ready for review November 6, 2025 16:48

cdransf requested a review from a team as a code owner November 6, 2025 16:48

cdransf requested review from graynorton and nikkimk November 6, 2025 16:49

cdransf self-assigned this Nov 6, 2025

cdransf added Status: Ready for review PR ready for review or re-review. and removed Status: WIP PR is a work in progress or draft labels Nov 6, 2025

cdransf force-pushed the ruben/playwright-a11y-testing branch from 9cb06e7 to 0545b8d Compare November 6, 2025 16:58

cdransf approved these changes Nov 6, 2025

View reviewed changes

cdransf force-pushed the ruben/playwright-a11y-testing branch from 7134b97 to 0545b8d Compare November 6, 2025 17:06

github-actions bot reviewed Nov 6, 2025

View reviewed changes

package.json Show resolved Hide resolved

package.json Show resolved Hide resolved

cdransf marked this pull request as draft November 6, 2025 17:12

cdransf removed the Status: Ready for review PR ready for review or re-review. label Nov 6, 2025

cdransf added the blocked Ticket or PR is blocked for some reason, eg another PR needs to go in first label Nov 6, 2025

cdransf self-requested a review November 6, 2025 17:14

rubencarvalho marked this pull request as ready for review November 6, 2025 17:26

fix(a11y-testing-spike): address test failures

cf00510

cdransf force-pushed the ruben/playwright-a11y-testing branch from 0545b8d to cf00510 Compare November 6, 2025 17:26

cdransf marked this pull request as draft November 6, 2025 17:27

cdransf marked this pull request as ready for review November 6, 2025 17:35

cdransf approved these changes Nov 6, 2025

View reviewed changes

cdransf merged commit 43c8eea into main Nov 6, 2025
22 checks passed

cdransf deleted the ruben/playwright-a11y-testing branch November 6, 2025 17:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(testing): add playwright setup #5860

feat(testing): add playwright setup #5860

Uh oh!

rubencarvalho commented Nov 6, 2025 •

edited by cdransf

Loading

Uh oh!

changeset-bot bot commented Nov 6, 2025 •

edited

Loading

Uh oh!

rubencarvalho Nov 6, 2025

Uh oh!

cdransf Nov 6, 2025

Uh oh!

Uh oh!

rubencarvalho Nov 6, 2025 •

edited

Loading

Uh oh!

cdransf Nov 6, 2025

Uh oh!

rubencarvalho Nov 6, 2025

Uh oh!

cdransf Nov 6, 2025

Uh oh!

rubencarvalho commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(testing): add playwright setup #5860

feat(testing): add playwright setup #5860

Uh oh!

Conversation

rubencarvalho commented Nov 6, 2025 • edited by cdransf Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

1. ARIA Snapshot Testing

2. aXe-core WCAG Validation

Test Coverage

Key Features

Usage

Files Added/Modified

Motivation and Context

Technical Decisions

Related Issue(s)

Author's Checklist

Reviewer's Checklist

Questions for Discussion

Manual Testing

Uh oh!

changeset-bot bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

rubencarvalho Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

cdransf Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rubencarvalho Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdransf Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

rubencarvalho Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

cdransf Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

rubencarvalho commented Nov 6, 2025

Uh oh!

github-actions bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Branch Preview

🔍 Visual Regression Test Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rubencarvalho commented Nov 6, 2025 •

edited by cdransf

Loading

changeset-bot bot commented Nov 6, 2025 •

edited

Loading

rubencarvalho Nov 6, 2025 •

edited

Loading

github-actions bot commented Nov 6, 2025 •

edited

Loading