Upgrade Doc Detective and fix test timing issues

[JakeSCahill]

Description

This PR upgrades Doc Detective to version 3.6.0 and implements workarounds for timing issues that cause test failures when finding UI elements after navigation.

Upgraded Doc Detective

This upgrade happened in stages due to issues discovered and reported:

Initial upgrade to v3.5.0

• Encountered critical bugs:
  • detectSteps config option was being ignored (#159)
  • CSS selectors with data-testid attributes were being parsed incorrectly

Upgrade to v3.5.1

• Fixed the detectSteps bug
• Still had issues with data-testid selector parsing

Final upgrade to v3.6.0

• Fixed XML attribute parsing bug that was breaking data-testid selectors
• All reported issues resolved
• Tests now working with proper selector support

Related upstream fixes:

• Bug Report: detectSteps config being ignored, causing step parsing failures doc-detective/doc-detective#159 (detectSteps bug)
• The data-testid parsing issue was fixed in the v3.6.0 release

Updated dependencies:

• package.json - Doc Detective version
• package-lock.json - All related dependencies

Added navigation and screenshot steps

Added explicit goTo and screenshot steps to ensure proper page rendering before element lookups:

Security Section (modules/get-started/pages/quick-start.adoc:277-283)

• Added goTo step to navigate to /security/users
• Added screenshot step to capture security page state
• This ensures the "Create user" button is findable

Topics Section (modules/get-started/pages/quick-start.adoc:419-426)

• Added goTo step to navigate to /topics
• Added screenshot step to capture topics page state
• This ensures the "Show internal topics" checkbox is findable

Improved test reliability

Added additional wait steps in several locations to improve test stability:

• After login page navigation
• After dropdown interactions
• After page transitions

Issues encountered

Upgrade issues (Resolved in v3.6.0)

During the upgrade process, we discovered and reported critical bugs that prevented tests from running:

1. detectSteps config ignored (#159)

   • Configuration to disable automatic step detection wasn't being respected
   • Fixed in v3.5.1

2. data-testid selector parsing broken

   • CSS selectors containing data-testid attributes were being incorrectly parsed
   • The XML attribute parsing function was too aggressive, stripping parts of the selector
   • Fixed in v3.6.0

Remaining timing issues (workaround required)

Even with v3.6.0's improved dynamic page load resolution, we discovered timing issues where Doc Detective fails to find elements even after using goTo + wait steps. The elements are present in the DOM with correct selectors, but the find step times out.

Root cause: The screenshot step appears to trigger browser rendering/paint cycles that make elements discoverable, suggesting Doc Detective may not be waiting for the page to be fully interactive after navigation, despite the v3.5.0 improvements to network traffic monitoring.

Workaround: Adding a screenshot step before problematic find operations resolves the issue. While not ideal, this workaround:

• Provides visual documentation of page states
• Ensures tests pass reliably
• Can be removed once the upstream issues are resolved

Related upstream issues reported:

• Bug report: Timing issue where elements not found despite goTo + wait steps doc-detective/doc-detective#165 (Element timing issue)
• Bug report: maxVariation Is interpreted as percent instead of fraction doc-detective/doc-detective#164 (Navigation reliability)

Testing

All tests now pass successfully:

npm run test-quickstart

Results:

• Total steps: 83
• Passed: 77
• Failed: 0
• Status: ✅ All items passed

Future work

Once the upstream Doc Detective issues are resolved, we can:

1. Remove the workaround screenshot steps from non-documentation locations
2. Potentially reduce wait times if rendering becomes more predictable
3. Simplify test sequences by relying on Doc Detective's built-in timing mechanisms

Page previews

Checks

• New feature
• Content gap
• Support Follow-up
• Small fix (typos, links, copyedits, etc)

[netlify]

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	b4ac581
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/692ef769fc3def0008463d9f
😎 Deploy Preview	https://deploy-preview-1495--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request updates test configuration and automation selectors across documentation testing files. Changes include updating the doc-detective dependency from version 3.1.0 to 3.6.0, migrating element selectors from CSS-based and direct text attributes to test ID-based attributes (e.g., auth-username-input, auth-submit), adjusting wait timeouts and screenshot comparison parameters (maxVariation), changing the browser configuration from Firefox to Chrome, and reordering test steps to align with the new selector approach in both the quick-start and authentication documentation tests.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

• quick-start.adoc: Large number of selector updates and timing adjustments; verify consistency of new test ID usage and timeout values across all steps
• Timing and waits in quick-start.adoc: Multiple additions and extensions of wait steps (2000ms, 5000ms); ensure timing changes are justified and don't introduce flakiness
• Screenshot configuration changes: Verify maxVariation parameters are correctly set for image comparison tolerance across authentication.adoc and quick-start.adoc
• Browser change in .doc-detective.json: Confirm Firefox-to-Chrome switch is intentional and compatible with all test steps

Possibly related PRs

• redpanda-data/docs#1148: Modifies doc-detective configuration and screenshot maxVariation parameters in the same authentication test artifacts
• redpanda-data/docs#1398: Updates UI test interaction timing, waits, and selectors in modules/get-started/pages/quick-start.adoc

Suggested reviewers

• paulohtb6
• kbatuigas

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main changes: upgrading Doc Detective and fixing test timing issues, which are the core objectives of this PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The pull request description is comprehensive and well-structured, exceeding template requirements with detailed context about the upgrade journey, issues encountered, workarounds implemented, and testing results.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch doc-detective

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (2)

modules/get-started/pages/quick-start.adoc (2)

238-242: Documentation tab step uses very long 15000ms wait; clarify if this is a workaround for animation/rendering delay.

The 15-second wait on line 241 (after clicking the Documentation tab at line 240) is substantially longer than other waits in the flow. Per the PR objectives, this appears to be a workaround for timing issues. Verify that this delay is necessary and document the underlying cause (animation duration, data loading, browser rendering) so it can be revisited when upstream doc-detective issues (#159, #164, #165) are resolved.

Consider adding an inline comment explaining the 15000ms wait duration and linking to the related upstream doc-detective issues mentioned in the PR summary.

---

1-687: Overall test file shows partial migration to doc-detective v3.6.0 patterns with strategic timing workarounds.

The quick-start test file demonstrates the intended migration (elementTestId selectors in login flow) and timing workarounds (2000–15000ms waits), consistent with the PR objectives. However, the file exhibits selector format inconsistency: the login flow (lines 180–182) uses the new elementTestId approach, while other sections (topic filter, security) continue using legacy CSS selectors with data-testid. Given that tests currently pass (77/83 steps per PR), consider documenting whether legacy selectors are intentionally retained due to missing test IDs in targeted elements or should be systematically migrated in a follow-up refactor.

Consider creating an internal tracking note or issue to systematically migrate remaining CSS selectors to elementTestId format once all UI elements have test IDs assigned, to improve maintainability and alignment with doc-detective v3.6.0 best practices.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

• Jira integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 328b1f2 and 36c9b55.

⛔ Files ignored due to path filters (14)

• modules/console/images/audit-logs.png is excluded by !**/*.png
• modules/console/images/broker-overview.png is excluded by !**/*.png
• modules/console/images/degraded-cluster.png is excluded by !**/*.png
• modules/console/images/internal-topics.png is excluded by !**/*.png
• modules/console/images/js-filter.png is excluded by !**/*.png
• modules/console/images/license.png is excluded by !**/*.png
• modules/console/images/overview.png is excluded by !**/*.png
• modules/console/images/schema-reg.png is excluded by !**/*.png
• modules/console/images/security.png is excluded by !**/*.png
• modules/console/images/topic-documentation.png is excluded by !**/*.png
• modules/console/images/topic.png is excluded by !**/*.png
• modules/console/images/transforms.png is excluded by !**/*.png
• modules/console/images/user.png is excluded by !**/*.png
• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (5)

• modules/console/pages/config/security/authentication.adoc (1 hunks)
• modules/get-started/pages/quick-start.adoc (8 hunks)
• package.json (1 hunks)
• tests/setup-tests/.doc-detective.json (1 hunks)
• tests/setup-tests/cloud/cloud-log-in.json (2 hunks)

🔇 Additional comments (6)

tests/setup-tests/.doc-detective.json (1)

11-11: Browser switch from Firefox to Chrome aligns with doc-detective v3.6.0 upgrade.

This change supports the updated test infrastructure. Ensure any browser-specific behavior (rendering, timing, element discovery) has been validated, especially given the timing workarounds added elsewhere in this PR.

tests/setup-tests/cloud/cloud-log-in.json (1)

15-17: Selector migration to elementAttribute is consistent with doc-detective v3.6.0.

The changes migrate from CSS selectors to semantic attribute-based targeting (name, data-provider, type), which aligns with the broader PR migration pattern and should improve selector robustness.

Also applies to: 29-31, 39-41

modules/get-started/pages/quick-start.adoc (2)

189-191: Line 189 lacks elementTestId while similar lines include it.

Line 189 uses only selector and elementTestId, missing the elementTestId field that is present in lines 181–182. Line 180 uses only elementTestId (most direct), while lines 181–182 combine both selector and elementTestId. If the "View" button has a test ID available, line 189 should be updated to include elementTestId for consistency and robustness; otherwise, document why this selector cannot be migrated.

---

283-283: Line 283's timeout of 100000ms is consistent with non-UI operations in the file and not unusual; however, line 289 has a structural JSON error with the "variables" key outside the "find" object.

Line 283 uses a 100-second timeout that matches similar timeouts for time-intensive operations elsewhere in the file (e.g., line 136's docker pull, line 137's docker compose up). This is reasonable for operations known to be slow and is not "dramatically higher" than precedent in the file.

However, line 289 contains a malformed JSON structure where the "variables" key appears at the same level as "find" rather than nested within it. This breaks the expected step object structure. Correct the syntax to place "variables" appropriately within the step definition.

Likely an incorrect or invalid review comment.

modules/console/pages/config/security/authentication.adoc (1)

18-18: No action required — maxVariation value is correct and consistent.

The maxVariation parameter uses a 0–1 scale (not a percentage), where the default is 0.05. The value 0.10 is valid, within specification, and already standardized across all screenshot steps in the documentation. This setting allows up to 10% visual variation when comparing captured images to references, which is more permissive than the default 0.05 and will not cause test flakiness.

package.json (1)

24-24: Verify doc-detective v3.6.0 compatibility against internal PR documentation.

Version 3.6.0 is published and exists on npm with dependencies on doc-detective-core and doc-detective-common at matching version. However, public release notes for v3.6.0 are not available to confirm the specific fixes for data-testid parsing and XML attribute issues mentioned in the PR. The caret constraint (^3.6.0) is appropriate for safe future patches. Verify the claimed bug fixes and any breaking changes directly against internal PR documentation or release notes.