DOC-1667 Document Shadow Link in Cloud #1491

micheleRP · 2025-11-25T00:09:26Z

Description

This pull request single sources and conditionalizes Shadowing content for Cloud.

It also restructures the configuration page, moving conceptual content into the overview & moving Create shadow link section toward the top of the config page.

(Single sourcing for this in cloud-docs repo is done in redpanda-data/cloud-docs#462. That one will merge Dec 12, to make Shadowing visible in Cloud docs.)

Added // tag::single-source[] and // end::single-source[] markers to support reuse and conditional rendering across cloud and self-hosted documentation. [1] [2] [3] [4] [5]
Introduced ifdef::env-cloud[] and ifndef::env-cloud[] blocks for environment-specific notes, requirements, and tips—such as licensing, authentication, and cluster property configuration. [1] [2] [3]
Added cloud-specific tips for shadow link management and clarified the flexibility of shadow link creation and management in Redpanda Cloud. [1] [2]

Resolves https://redpandadata.atlassian.net/browse/DOC-1667
Review deadline: Dec 10

Page previews

What's New in Redpanda Cloud
Shadowing in Cloud
rpk shadow in Cloud

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

netlify · 2025-11-25T00:09:32Z

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	`2b79cbc`
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/692f5c342eb2130008dd03c6
😎 Deploy Preview	https://deploy-preview-1491--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

coderabbitai · 2025-11-25T00:09:32Z

📝 Walkthrough

Walkthrough

This PR implements environment-specific documentation branching for disaster recovery shadowing features. It updates the Antora playbook to reference a feature branch for cloud-docs content, then modifies multiple shadowing documentation files to include conditional content blocks that differentiate between cloud and non-cloud environments. Environment-specific sections are gated using ifndef::env-cloud[] and ifdef::env-cloud[] directives, and single-source region markers delineate content boundaries. A new cloud-specific tip file is introduced, cross-reference links are corrected from deploy: to manage: module paths, and minor terminology and formatting adjustments are applied throughout.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Feature branch reference in local-antora-playbook.yml requires verification for correctness and merge strategy
Multiple interconnected documentation files with paired conditional blocks and single-source markers need validation for proper closure and nesting
Cross-reference link updates (deploy: → manage:) should be verified for accuracy across all affected files
Content rewording and terminology standardization (e.g., capitalization changes, metric label lowercase adjustments) should be checked for consistency
Ensure conditional content is appropriately present/absent in each environment branch

Possibly related PRs

PR #1481 — Modifies the same disaster-recovery shadowing documentation files and adjusts cross-reference targets to manage:disaster-recovery/...
PR #1329 — Adds/repairs single-source tag regions, gates license content with env-cloud conditionals, and fixes cross-repo links in similar fashion
PR #1478 — Modifies the same documentation surface (Antora playbook and multiple shadowing docs with environment-conditional markup changes)

Suggested reviewers

mattschumpert
paulohtb6
Feediver1

Pre-merge checks and finishing touches

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly refers to the main objective of the PR: documenting shadow links in cloud with comprehensive changes including single-sourcing, conditional blocks, and cloud-specific content.
Linked Issues check	✅ Passed	The PR comprehensively addresses the primary objective of documenting shadow links in cloud through single-sourcing markers, environment-specific conditionals, cloud-specific tips, and updated references DOC-1667.
Out of Scope Changes check	✅ Passed	All changes are directly related to the PR's stated objective of single-sourcing and conditionalizing shadowing content for cloud. The branch change in local-antora-playbook.yml supports building documentation from the correct cloud source.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The pull request description follows the template with a clear description section, relevant JIRA ticket link, review deadline, page previews, and checked appropriate categories.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch DOC-1667-Document-Shadow-Link-in-Cloud

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

treevon · 2025-12-01T23:31:04Z

In Monitoring, Under metrics, it would be great to include the detail that the metrics are available via the /public_metrics endpoint
"Shadowing provides comprehensive metrics to track replication performance and health via the [/public_metrics] endpoint":

treevon · 2025-12-01T23:33:33Z

In monitoring the Alert thresholds section should be renamed to Alert conditions
and the text below should have more of an explanation"
The following conditions indicate problems with shadowing and should have alerts configured for them

treevon · 2025-12-01T23:42:28Z

On Monitoring - The Status command options: feels duplicative, it doesn't have anything different than the previous command. Could we remove it?

treevon

Overall I like the new structure. I have some minor edits.

modules/manage/pages/disaster-recovery/shadowing/failover.adoc

treevon · 2025-12-01T23:39:41Z

modules/manage/pages/disaster-recovery/shadowing/setup.adoc

+The starting offset only affects **new shadow topics**. After a shadow topic exists and has data, changing this setting has no effect on that topic's replication.
 ====

 === Generate a sample configuration


Could we move this right under Set Up Shadowing? We should explain the configuration file before suggesting users create a shadow link using a configuration file.

Thanks @treevon! Please see the new structure here

modules/manage/pages/disaster-recovery/shadowing/setup.adoc

paulohtb6 · 2025-12-02T01:59:08Z

Reading this again with fresh eyes

docs/modules/manage/pages/disaster-recovery/shadowing/overview.adoc

Lines 213 to 221 in a68335b

    
           == Disaster readiness checklist 
        
           Before a disaster occurs, ensure you have: 
        
           [%interactive] 
        
           * [ ] Access to shadow cluster administrative credentials 
        
           * [ ] Shadow link names and configuration details, and networking documented 
        
           * [ ] Application connection strings for the shadow cluster prepared 
        
           * [ ] Tested failover procedures in a non-production environment

I think those lines are out of place for overview. Let's remove it. (cant create a suggestion because its not on the changed lines)

modules/manage/pages/disaster-recovery/shadowing/overview.adoc

modules/manage/pages/disaster-recovery/shadowing/setup.adoc

paulohtb6 · 2025-12-02T02:16:18Z

modules/manage/pages/disaster-recovery/shadowing/setup.adoc


 === Administrative access

 Superuser access is required on both clusters through xref:get-started:rpk/index.adoc[`rpk`], the Admin API, or xref:console:index.adoc[Redpanda Console] to create and manage shadow links.


What's the admin requirement for cloud? do we have superusers there too? @simon0191

For context

SM https://deploy-preview-1491--redpanda-docs-preview.netlify.app/current/manage/disaster-recovery/shadowing/setup/#administrative-access

Cloud https://deploy-preview-1491--redpanda-docs-preview.netlify.app/redpanda-cloud/manage/disaster-recovery/shadowing/setup/#prerequisites

paulohtb6 · 2025-12-02T02:18:15Z

modules/manage/pages/disaster-recovery/shadowing/setup.adoc


 **Avoid name conflicts:** If you plan to consume data from the shadow cluster, do not use the same consumer group names as those used on the source cluster. While this won't break shadow linking, it can impact your RPO/RTO because conflicting group names may interfere with offset replication and consumer resumption during disaster recovery.

 **Offset clamping:** When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition's HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster.


Suggested change

**Offset clamping:** When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition's HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster.

**Offset clamping:** When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the HWM of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition's HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster.

with glossary maybe we can say HWM for high water mark

simon0191 · 2025-12-02T11:41:51Z

modules/manage/pages/disaster-recovery/shadowing/monitor.adoc

-
-* **Shadow link state**: Overall operational state (`ACTIVE`)
-* **Individual topic states**: Current state of each replicated topic (`ACTIVE`, `FAULTED`, `FAILING_OVER`, `FAILED_OVER`)
+* **Shadow link state**: Overall operational state (`ACTIVE`).


This one can be ACTIVE or PAUSED

https://github.com/redpanda-data/redpanda/blob/dev/proto/redpanda/core/admin/v2/shadow_link.proto#L553-L559

@simon0191 I updated on Monitor page and also on Failover page. Please confirm that the description for PAUSED is correct.

modules/manage/pages/disaster-recovery/shadowing/monitor.adoc

simon0191 · 2025-12-02T11:45:59Z

modules/manage/pages/disaster-recovery/shadowing/overview.adoc

+The architecture follows an active-passive pattern. The source cluster processes all production traffic while the shadow cluster remains in read-only mode, continuously receiving updates. If a disaster occurs, you can failover the shadow topics using the 
+ifndef::env-cloud[Admin API] 
+ifdef::env-cloud[Data Plane API] 
+or `rpk`, making them fully writable. At that point, you can redirect your applications to the shadow cluster, which becomes the new production cluster.


I think we should warn the user about the potential for creating a split brain scenario if they don't ensure that all their clients are reconfigured to point to the shadow cluster.

I added a note in overview and in failover. (It's included as a step in the failover runbook.)

Co-authored-by: Paulo Borges <[email protected]>

micheleRP · 2025-12-02T21:43:02Z

Merging this now. Kat will then create a new PR to add Cloud API info and any other updates necessary for Cloud with rpk & UI. Single sourcing for this in cloud-docs repo is done in redpanda-data/cloud-docs#462. That one will merge Dec 12, to make Shadowing visible in Cloud docs. Preview docs showing the updates from this PR in Cloud docs are available here.

micheleRP mentioned this pull request Nov 25, 2025

DOC-1621 Shadowing in Cloud redpanda-data/cloud-docs#462

Open

4 tasks

micheleRP force-pushed the DOC-1667-Document-Shadow-Link-in-Cloud branch from f533461 to 6b96f68 Compare November 25, 2025 19:05

micheleRP marked this pull request as ready for review November 25, 2025 20:03

micheleRP requested a review from a team as a code owner November 25, 2025 20:03

treevon suggested changes Dec 1, 2025

View reviewed changes

paulohtb6 reviewed Dec 2, 2025

View reviewed changes

modules/manage/pages/disaster-recovery/shadowing/overview.adoc Outdated Show resolved Hide resolved

paulohtb6 reviewed Dec 2, 2025

View reviewed changes

modules/manage/pages/disaster-recovery/shadowing/setup.adoc Outdated Show resolved Hide resolved

paulohtb6 reviewed Dec 2, 2025

View reviewed changes

modules/manage/pages/disaster-recovery/shadowing/setup.adoc Outdated Show resolved Hide resolved

paulohtb6 reviewed Dec 2, 2025

View reviewed changes

micheleRP requested a review from simon0191 December 2, 2025 02:21

paulohtb6 approved these changes Dec 2, 2025

View reviewed changes

simon0191 reviewed Dec 2, 2025

View reviewed changes

micheleRP added 11 commits December 2, 2025 12:59

DOC-1667 Document Shadow Link in Cloud

0228148

single source rpk shadow

bffbfea

conditionalize parts for Cloud

d3db493

add description for rpk shadow (for index page)

05bac83

minor edits

d3e108c

fix headings

9c5e166

update conditionalizing

7a6ffad

move conceptual sections from setup into overview

781937a

rearranging setup

6e8ac75

byoc/dedicated only

582e395

conditionalizing

614e5c5

micheleRP and others added 8 commits December 2, 2025 12:59

change Configure Failover to Failover

043e166

incorporate Trevor's review feedback

763f4da

minor edit

6c97ab7

move tip next to sample config file

841a5a6

remove disaster readiness checklist from overview

d8e0bbc

remove duplicated content

f48eefd

incorporate Simon's feedback

1d9c180

Update modules/manage/pages/disaster-recovery/shadowing/setup.adoc

aea6e82

Co-authored-by: Paulo Borges <[email protected]>

micheleRP force-pushed the DOC-1667-Document-Shadow-Link-in-Cloud branch from 3118ad6 to aea6e82 Compare December 2, 2025 19:59

revert playbook + minor edits

2b79cbc

micheleRP merged commit 2a5e33a into main Dec 2, 2025
5 checks passed

micheleRP deleted the DOC-1667-Document-Shadow-Link-in-Cloud branch December 2, 2025 21:44

coderabbitai bot mentioned this pull request Dec 2, 2025

add shadowing metrics #1496

Open

4 tasks


		=== Administrative access

		Superuser access is required on both clusters through xref:get-started:rpk/index.adoc[`rpk`], the Admin API, or xref:console:index.adoc[Redpanda Console] to create and manage shadow links.


		Avoid name conflicts: If you plan to consume data from the shadow cluster, do not use the same consumer group names as those used on the source cluster. While this won't break shadow linking, it can impact your RPO/RTO because conflicting group names may interfere with offset replication and consumer resumption during disaster recovery.

		Offset clamping: When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition's HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster.

DOC-1667 Document Shadow Link in Cloud #1491

DOC-1667 Document Shadow Link in Cloud #1491

Conversation

micheleRP commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Page previews

Checks

Uh oh!

netlify bot commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for redpanda-docs-preview ready!

Uh oh!

coderabbitai bot commented Nov 25, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Estimated code review effort

Possibly related PRs

Suggested reviewers

Pre-merge checks and finishing touches

Uh oh!

treevon commented Dec 1, 2025

Uh oh!

treevon commented Dec 1, 2025

Uh oh!

treevon commented Dec 1, 2025

Uh oh!

treevon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

paulohtb6 commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

micheleRP commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

micheleRP commented Nov 25, 2025 •

edited

Loading

netlify bot commented Nov 25, 2025 •

edited

Loading

coderabbitai bot commented Nov 25, 2025 •

edited by atlassian bot

Loading

paulohtb6 commented Dec 2, 2025 •

edited

Loading

micheleRP commented Dec 2, 2025 •

edited

Loading