Doc 1602: Cloud Topics (Beta) #1469
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the 📝 WalkthroughWalkthroughThis PR reorganizes Redpanda's topic-related documentation by creating a new Estimated code review effort🎯 2 (Simple) | ⏱️ ~15 minutes
Areas to review:
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
modules/develop/pages/manage-topics/config-topics.adoc (1)
92-116: Fix CLI flag typo in alter-config example.
The example uses a long dash beforeset. It should be--set(two hyphens) or the command fails.Apply:
- rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact + rpk topic alter-config [TOPICS...] --set cleanup.policy=compactAlso applies to: 105-111
🧹 Nitpick comments (2)
modules/develop/pages/manage-topics/index.adoc (1)
1-4: Nice addition. Consider brief “What’s here” bullets.
Add 2–3 bullets linking to child pages (config-topics, cloud-topics) to help orientation.modules/develop/pages/manage-topics/cloud-topics.adoc (1)
17-24: Beta feature guardrails look good; add cluster-level enable note if required.
If Cloud Topics require a cluster flag (for dev/testing), add a note referencing the relevant cluster property (for example, if any) or document that none is needed.Also applies to: 32-40
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (16)
modules/ROOT/nav.adoc(1 hunks)modules/console/pages/ui/edit-topic-configuration.adoc(1 hunks)modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc(1 hunks)modules/develop/pages/kafka-clients.adoc(1 hunks)modules/develop/pages/manage-topics/cloud-topics.adoc(1 hunks)modules/develop/pages/manage-topics/config-topics.adoc(2 hunks)modules/develop/pages/manage-topics/index.adoc(1 hunks)modules/develop/pages/produce-data/configure-producers.adoc(1 hunks)modules/get-started/pages/quick-start.adoc(1 hunks)modules/manage/pages/cluster-maintenance/compaction-settings.adoc(1 hunks)modules/manage/pages/cluster-maintenance/disk-utilization.adoc(1 hunks)modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc(1 hunks)modules/manage/partials/audit-logging.adoc(1 hunks)modules/reference/pages/properties/cluster-properties.adoc(1 hunks)modules/reference/pages/properties/topic-properties.adoc(2 hunks)modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc(1 hunks)
🧰 Additional context used
🧠 Learnings (5)
📓 Common learnings
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
📚 Learning: 2025-07-16T19:33:20.420Z
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
Applied to files:
modules/manage/pages/cluster-maintenance/compaction-settings.adocmodules/reference/pages/properties/cluster-properties.adocmodules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adocmodules/reference/pages/properties/topic-properties.adocmodules/manage/pages/cluster-maintenance/topic-property-configuration.adocmodules/console/pages/ui/edit-topic-configuration.adocmodules/get-started/pages/quick-start.adocmodules/develop/pages/kafka-clients.adocmodules/deploy/pages/redpanda/manual/production/dev-deployment.adoc
📚 Learning: 2025-05-07T01:06:00.937Z
Learnt from: kbatuigas
Repo: redpanda-data/docs PR: 1113
File: modules/manage/partials/iceberg/use-iceberg-catalogs.adoc:100-107
Timestamp: 2025-05-07T01:06:00.937Z
Learning: In AsciiDoc documentation for Redpanda, the syntax `+` and `--` around content blocks within a `[tabs]` section are valid AsciiDoc formatting elements for tabbed content. The `+` after a tab name (like `rpk::`) indicates that the following block belongs to that tab, and the `--` markers enclose the content for that tab. These are not diff artifacts and should not be removed.
Applied to files:
modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc
📚 Learning: 2025-08-25T21:00:26.626Z
Learnt from: micheleRP
Repo: redpanda-data/docs PR: 1334
File: modules/manage/partials/rbac-dp.adoc:93-98
Timestamp: 2025-08-25T21:00:26.626Z
Learning: In cloud documentation (env-cloud), Security is at the top level navigation, so ACL references should use `security:authorization/rbac/acl.adoc`. In self-managed documentation, Security is nested under Manage, so ACL references use `manage:security/authorization/acl.adoc`. The different xref paths in conditional blocks reflect these different navigation structures.
Applied to files:
modules/develop/pages/manage-topics/cloud-topics.adocmodules/manage/pages/cluster-maintenance/topic-property-configuration.adocmodules/ROOT/nav.adoc
📚 Learning: 2025-08-19T19:26:57.364Z
Learnt from: paulohtb6
Repo: redpanda-data/docs PR: 1324
File: modules/deploy/partials/high-availability.adoc:537-537
Timestamp: 2025-08-19T19:26:57.364Z
Learning: In AsciiDoc documentation, page aliases (`:page-aliases:`) should never be removed as they serve as old redirects to maintain backward compatibility when pages are moved or renamed. They preserve SEO and prevent broken links for users.
Applied to files:
modules/develop/pages/manage-topics/config-topics.adoc
🔇 Additional comments (16)
modules/manage/pages/cluster-maintenance/compaction-settings.adoc (1)
15-15: Verify the anchor#change-the-cleanup-policyexists in the reorganized target.This line references an anchor that should exist in the moved config-topics.adoc file. Based on the retrieved learnings, anchors for topic configuration properties and sections must be explicitly defined or auto-generated from section headers.
modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc (1)
11-11: LGTM.The cross-reference update correctly points to the reorganized manage-topics/config-topics.adoc path and maintains the same anchor reference for write caching configuration. The context and surrounding text remain appropriate.
modules/manage/partials/audit-logging.adoc (1)
436-436: LGTM.The cross-reference update correctly reflects the new manage-topics directory structure. Without an anchor fragment, this reference will point to the top-level page, which is appropriate for a general "Related topics" reference.
modules/manage/pages/cluster-maintenance/disk-utilization.adoc (1)
88-88: Verify anchor#delete-records-from-a-topicexists in target file.This anchor reference should correspond to a section in manage-topics/config-topics.adoc covering how to delete records from topics. Based on the PR's learnings about anchors, confirm this section (or an explicit anchor) exists in the reorganized target file.
modules/develop/pages/kafka-clients.adoc (1)
63-63: Verify module prefix change is intentional.This cross-reference is unique among the PR changes: it shifts the module prefix from
get-started:todevelop:in addition to restructuring the path tomanage-topics/config-topics.adoc. This suggests config-topics.adoc has been moved to the develop module (rather than just reorganized within it).Confirm this module relocation aligns with the PR's overall reorganization strategy and that no residual reference in the get-started module should exist.
modules/get-started/pages/quick-start.adoc (1)
457-457: Verify anchor#choose-the-replication-factorexists in target file.This anchor reference is expected to correspond to a section in manage-topics/config-topics.adoc that explains replication factor selection. Confirm this section or an explicit anchor with this ID exists in the reorganized target file to prevent broken links.
modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc (1)
223-223: LGTM.The cross-reference update correctly points to the new manage-topics/config-topics.adoc path. This is a straightforward top-level page reference without an anchor, appropriate for the "Related topics" section context.
modules/develop/pages/produce-data/configure-producers.adoc (1)
77-77: Anchor verification complete—xref is valid.The xref fragment
#configure-write-cachingwill be auto-generated from the section heading "=== Configure write caching" at line 117 of the target file. AsciiDoc/Antora automatically generates anchors from headings using lowercase text with hyphens replacing spaces. The reorganized cross-reference is valid and will not produce broken links.modules/console/pages/ui/edit-topic-configuration.adoc (1)
22-22: LGTM — updated xref path is correct.modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc (1)
13-13: LGTM — write-caching xref now targets the new location.modules/develop/pages/manage-topics/config-topics.adoc (1)
4-4: Good — alias preserves old URL (back-compat).modules/ROOT/nav.adoc (1)
39-41: LGTM — nav reflects new Manage topics hub.modules/reference/pages/properties/topic-properties.adoc (1)
488-489: LGTM — write-caching cross-ref updated to new section.modules/develop/pages/manage-topics/cloud-topics.adoc (2)
41-42: Strong GA migration statement — confirm with PM.
“Throw away any cluster…” is severe. If true for Public Beta, add context (applies only to beta clusters) and guidance on migration; otherwise soften.
14-14: Avoid exact latency claim unless documented.
“The difference in response time is typically 500ms” — either add a “typical, workload-dependent” qualifier with a citation or remove the number.modules/reference/pages/properties/cluster-properties.adoc (1)
6549-6549: LGTM — related topics link updated to the new anchor. Verification complete.Verification confirms:
- The xref at line 6549 in cluster-properties.adoc correctly points to
develop:manage-topics/config-topics.adoc#configure-write-caching- The target anchor exists in config-topics.adoc as an auto-generated anchor from the section heading "=== Configure write caching" (line 117)
- No lingering old-style references to
develop:config-topics.adoc(without the manage-topics path) remain in the codebase
Feediver1
force-pushed
the
DOC-1602
branch
2 times, most recently
from
4645ef8 to
90dc1e8
Compare
| ==== | ||
| endif::[] | ||
|
|
||
| Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible. While standard Kafka xref:config-topics.adoc[topics] are ideal for latency-sensitive, high-throughput workloads (for example, for audit logs), Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. |
There was a problem hiding this comment.
"Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] using local storage or tiered storage are ideal for latency-sensitive, lower throughput workloads, Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-zone networking charges) are a major consideration and can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds or just development environments where streaming latency isn't critical. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, , ADLS, GCS, MinIO etc) as the primary mechanism for storing topic data, eliminating over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. Redpanda also allows you to mix and match cloud topics workloads with low latency workloads in the same cluster, and will (at GA time) allow you to migrate existing local or tiered storage topics to become cloud topics.
There was a problem hiding this comment.
@Feediver1 re-watching Noah's demo, I think it does make sense to include some of these rough cost savings numbers somewhere (though we should be clear this is just replication costs), e.g. something like this:
"The infrastructure cost savings with Cloud Topics can be quite dramatic. By way of example, in one cloud environment tested by Redpanda, the networking costs of data replication to run a single 100MBps workload were reduced from over $10,000 per month to ~$80 per month. While this doesn't represent 100% of total operating costs, and noting that kafka batch sizes can affect this cost reduction slightly, at high throughputs this savings still accounts for the vast majority of cloud provider expense, reducing the total infrastructure cost of running Kafka workloads by an order of magnitude or more."
There was a problem hiding this comment.
No can do on "Redpanda also allows you to mix and match cloud topics workloads with low latency workloads in the same cluster, and will (at GA time) allow you to migrate existing local or tiered storage topics to become cloud topics."
We don't ever pre-announce feature updates in the docs. This point can be added when it is in there at GA time.
There was a problem hiding this comment.
This intro is now quite verbose and overly complex...
There was a problem hiding this comment.
I think it does make sense to include some of these rough cost savings numbers somewhere (though we should be clear this is just replication costs)
I had advised against this as it felt like technical documentation not marketing material, so that's on me. I don't actually care either way.
There was a problem hiding this comment.
allow you to migrate existing local or tiered storage topics to become cloud topics.
Also I don't think this has been discussed explicitly as something we will support in GA.
There was a problem hiding this comment.
@dotnwat definitely that's been my understanding with Piyush. But wrt these docs we can just leave it out
doc review feedback Co-authored-by: Michele Cyran <[email protected]>
| rpk cluster license info | ||
| ---- | ||
|
|
||
| If you intend to use Docker Compose while working with Cloud Topics, see the https://docs.docker.com/compose/install/[Docker Compose installation documentation^]. To verify you have it installed, run: |
There was a problem hiding this comment.
Should this be a separate bulleted item under Prereqs? If not (if it should instead be nested under the enterprise license), then add + in the line breaks, like in the above suggestion
|
|
||
| Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. | ||
|
|
||
| Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, ADLS,GCS, MinIO) as the primary mechanism for backing up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. |
There was a problem hiding this comment.
There was a problem hiding this comment.
@Feediver1 this describes the architecture a bit (what CS was asking for) put this paragraph wherever you think it makes the most sense.
"With cloud topics, Data from the client is not acknowledged until it is uploaded to cloud storage, maintaining durability in the face of infrastructure failures, but resulting in an increase in both produce latency and end-to-end-latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency), but in general, you should expect latencies roughly in the range of 250ms-1s."
(maybe put this part in an info box @Feediver1 )
*Tip: Given the architecture and latency profile of cloud topics, a producer configuration of ACKs=all is always recommended, as other configurations (e.g. acks=0/1) will not yield a lower produce latency.
|
|
||
| Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. | ||
|
|
||
| Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, ADLS,GCS, MinIO) as the primary mechanism for backing up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. |
There was a problem hiding this comment.
@Feediver1 maybe 'eliminates over 90% replication traffic across AZs/regions in multi-az or multi-region clusters'.
we should keep the latency stuff and then add comments on produce latency, maybe this is a new paragraph.
| ---- | ||
|
|
||
| You can make a topic a Cloud Topic only at topic creation time. | ||
|
|
There was a problem hiding this comment.
@Feediver1 maybe here is where we add the 'pro tip' for Producer Acks setting.
Co-authored-by: Michele Cyran <[email protected]>
Co-authored-by: Michele Cyran <[email protected]>
doc review feedback Co-authored-by: Michele Cyran <[email protected]>
Description
Resolves https://redpandadata.atlassian.net/browse/DOC-1602
Review deadline:
DO NOT MERGE
Page previews
Cloud Topics << URL to share with beta users
Checks