changefeedccl: protect lagging tables with its own timestamp #149616

aerfrei · 2025-07-07T19:16:53Z

Before this change, changefeeds watching multiple tables
would protect all watched tables and system tables with a
single protected timestamp (pts) record. All tables would
be protected back to the highwater for the changefeed.

In the event where one table lags behind the others, this
would result in preventing more garbage collection than
needed for many tables. This would become more of an issue
when we introduce DB-level feeds because they will be
implemented with multi-table feeds.

We now protect lagging tables separately. We use per table
highwaters to identify tables who are lagging behing the
most advanced table by more than the new
changefeed.protect_timestamp_bucketing_interval cluster
setting. All tables at least that far behind will be protected
by their own pts record.

Epic: CRDB-1421
Fixes: #147785

Release note (performance): Multi-table changefeeds will protect
each lagging user table from garbage collection with its own record.
This will improve performance of multi-table changefeeds including
upcoming db-level changefeeds by allowing more garbage collection.

cockroach-teamcity · 2025-07-07T19:17:11Z

This change is

blathers-crl · 2025-07-18T19:50:16Z

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

pkg/ccl/changefeedccl/changefeedbase/settings.go

pkg/jobs/jobspb/jobs.proto

msbutler · 2025-08-04T20:35:11Z

adding @dt as a reviewer while i'm out on pto

andyyang890

This PR is starting to look really good!

pkg/ccl/changefeedccl/changefeed_processors.go

pkg/cmd/roachtest/tests/cdc.go

pkg/ccl/changefeedccl/changefeed_test.go

andyyang890

Prod code looks good, have some comments for the test code

pkg/cmd/roachtest/tests/cdc.go

pkg/ccl/changefeedccl/protected_timestamps_test.go

pkg/ccl/changefeedccl/changefeed_test.go

pkg/ccl/changefeedccl/protected_timestamps_test.go

Before this change, changefeeds watching multiple tables would protect all watched tables and system tables with a single protected timestamp (pts) record. All tables would be protected back to the highwater for the changefeed. In the event where one table lags behind the others, this would result in preventing more garbage collection than needed for many tables. This would become more of an issue when we introduce DB-level feeds because they will be implemented with multi-table feeds. We now protect lagging tables separately. We use per table highwaters to identify tables who are lagging behing the most advanced table by more than the new changefeed.protect_timestamp_bucketing_interval cluster setting. All tables at least that far behind will be protected by their own pts record. Epic: CRDB-1421 Fixes: cockroachdb#147785 Release note (performance): Multi-table changefeeds will protect each lagging user table from garbage collection with its own record. This will improve performance of multi-table changefeeds including upcoming db-level changefeeds by allowing more garbage collection.

andyyang890

Nice work! Thanks for your patience with all the feedback

aerfrei · 2025-09-02T17:41:00Z

bors r=andyyang890

craig · 2025-09-02T17:41:08Z

👎 Rejected by code reviews

This is stale, Andy's looked at it more recently

aerfrei · 2025-09-02T17:59:09Z

bors r=andyyang890

craig · 2025-09-02T19:36:53Z

Build succeeded:

commit 2214ab5 Author: Andy Yang <[email protected]> Date: Tue Sep 2 16:34:36 2025 -0400 break dependency from changefeedpb to parser commit 820a24b Author: Andy Yang <[email protected]> Date: Tue Sep 2 09:47:04 2025 -0400 fix sinkless issue commit 6bc3449 Author: Andy Yang <[email protected]> Date: Fri Aug 29 16:48:18 2025 -0400 fixup! attempt to fix partitioner commit b988450 Author: Andy Yang <[email protected]> Date: Fri Aug 29 16:36:21 2025 -0400 attempt to fix partitioner commit 0189f6b Author: Andy Yang <[email protected]> Date: Fri Aug 29 15:54:49 2025 -0400 wrap table partitioner errors commit defca28 Author: Andy Yang <[email protected]> Date: Fri Aug 22 17:34:00 2025 -0400 restore per-table data commit c63ae2f Author: Andy Yang <[email protected]> Date: Mon Aug 18 15:18:01 2025 -0400 add resolved tables to specs commit f4a53b3 Author: Andy Yang <[email protected]> Date: Mon Aug 18 14:47:32 2025 -0400 add protobuf commit 124989a Author: Andy Yang <[email protected]> Date: Wed Aug 27 20:30:59 2025 -0400 changefeedccl/resolvedspan: update table ID partitioner This patch updates the table ID partitioner so that it's able to handle spans where the end key is the next table's start key (e.g. full table spans like `/Table/1{0-1}`). Release note: None commit c1e847b Merge: 3693763 021bb15 8de4c80 Author: craig[bot] <[email protected]> Date: Tue Sep 2 18:34:48 2025 +0000 Merge cockroachdb#149616 cockroachdb#152190 149616: changefeedccl: protect lagging tables with its own timestamp r=andyyang890 a=aerfrei Before this change, changefeeds watching multiple tables would protect all watched tables and system tables with a single protected timestamp (pts) record. All tables would be protected back to the highwater for the changefeed. In the event where one table lags behind the others, this would result in preventing more garbage collection than needed for many tables. This would become more of an issue when we introduce DB-level feeds because they will be implemented with multi-table feeds. We now protect lagging tables separately. We use per table highwaters to identify tables who are lagging behing the most advanced table by more than the new changefeed.protect_timestamp_bucketing_interval cluster setting. All tables at least that far behind will be protected by their own pts record. Epic: CRDB-1421 Fixes: cockroachdb#147785 Release note (performance): Multi-table changefeeds will protect each lagging user table from garbage collection with its own record. This will improve performance of multi-table changefeeds including upcoming db-level changefeeds by allowing more garbage collection. 152190: contention: periodically log resolved contention events r=alyshanjahani-crl a=alyshanjahani-crl This commit adds logging to the flushAndResolve method in the contention event store. This method is called periodically at an interval defined by the (non-public) cluster setting sql.contention.event_store.resolution_interval. By logging aggregated contention event information, users now have the ability to find which waiting statements/txns are experiencing high contention through logs (previously this was only available via insights or SQL activity). Additionally, this aggregated event information contains the blocking txn and contended key - information that was previously only available via querying crdb_internal.transaction_contention_events. The logs are aggregated by a composite identifier of (waiting stmt, waiting txn, blocking txn, key), and the total contention time observed for that combination. They are structured logs. For example: ``` {"Timestamp":1756224167482848000,"EventType":"aggregated_contention_info", "WaitingStmtFingerprintId":"\\x000000000000007b", "WaitingTxnFingerprintId":"\\x00000000000001c8", "BlockingTxnFingerprintId":"\\x0000000000000315", "ContendedKey":"test-key-1","Duration":300000000} ``` Since this information is available via logs, it provides a solution for users who desire more persistence of contention event information. Part of: https://cockroachlabs.atlassian.net/browse/CRDB-53592 Release note (ops change): Adds logging of contention events. Co-authored-by: Aerin Freilich <[email protected]> Co-authored-by: Alyshan Jahani <[email protected]> commit 3693763 Merge: 3aad29a 16bbe95 59221d2 7f054a0 Author: craig[bot] <[email protected]> Date: Tue Sep 2 17:58:20 2025 +0000 Merge cockroachdb#152848 cockroachdb#152850 cockroachdb#152856 152848: release: fix docker image verification with telemetry disabled r=celiala a=rail This commit fixes the verification of the FIPS docker image when telemetry is disabled. Epic: none Release note: none 152850: changefeedccl: ignore context canceled error in tests r=andyyang890 a=KeithCh Context canceled errors is causing flakes in tests, and should be ignored. Fixes: cockroachdb#151726 cockroachdb#152510 Release note: None 152856: roachtest: create required test schemas for SQLAlchemy test suite r=rafiss a=rafiss fixes cockroachdb#152780 fixes cockroachdb#152779 fixes cockroachdb#152778 fixes cockroachdb#152775 fixes cockroachdb#152773 Release note: None Co-authored-by: Rail Aliiev <[email protected]> Co-authored-by: Keith Chow <[email protected]> Co-authored-by: Rafi Shamim <[email protected]> commit 59221d2 Author: Keith Chow <[email protected]> Date: Tue Sep 2 08:50:05 2025 -0400 changefeedccl: ignore context canceled error in tests context canceled errors is causing flakes in tests, and should be ignored. Fixes: cockroachdb#151726 cockroachdb#152510 Release note: None commit 3aad29a Merge: d8d00d8 ef18807 Author: craig[bot] <[email protected]> Date: Tue Sep 2 17:13:23 2025 +0000 Merge cockroachdb#152854 152854: backup: remove endtime filter from GetBackupTreeIndexMetadata r=msbutler a=kev-cao This patch removes the `endTime` parameter from `GetBackupTreeIndexMetadata`. The filtering will already be done by `ValidateEndTimeAndTruncate` and so is unnecessary. The addition of this filtering results in a change in the error messages when an invalid end time is provided, so for the sake of simplicity, it's better to remove the parameter. Epic: None Release note: None Co-authored-by: Kevin Cao <[email protected]> commit 7f054a0 Author: Rafi Shamim <[email protected]> Date: Tue Sep 2 12:04:17 2025 -0400 roachtest: create required test schemas for SQLAlchemy test suite Release note: None commit 021bb15 Author: Aerin Freilich <[email protected]> Date: Fri Jul 18 16:00:01 2025 -0400 changefeedccl: protect lagging tables with its own timestamp Before this change, changefeeds watching multiple tables would protect all watched tables and system tables with a single protected timestamp (pts) record. All tables would be protected back to the highwater for the changefeed. In the event where one table lags behind the others, this would result in preventing more garbage collection than needed for many tables. This would become more of an issue when we introduce DB-level feeds because they will be implemented with multi-table feeds. We now protect lagging tables separately. We use per table highwaters to identify tables who are lagging behing the most advanced table by more than the new changefeed.protect_timestamp_bucketing_interval cluster setting. All tables at least that far behind will be protected by their own pts record. Epic: CRDB-1421 Fixes: cockroachdb#147785 Release note (performance): Multi-table changefeeds will protect each lagging user table from garbage collection with its own record. This will improve performance of multi-table changefeeds including upcoming db-level changefeeds by allowing more garbage collection. commit ef18807 Author: Kevin Cao <[email protected]> Date: Tue Sep 2 11:47:45 2025 -0400 backup: remove endtime filter from GetBackupTreeIndexMetadata This patch removes the `endTime` parameter from `GetBackupTreeIndexMetadata`. The filtering will already be done by `ValidateEndTimeAndTruncate` and so is unnecessary. The addition of this filtering results in a change in the error messages when an invalid end time is provided, so for the sake of simplicity, it's better to remove the parameter. Epic: None Release note: None commit 16bbe95 Author: Rail Aliiev <[email protected]> Date: Tue Sep 2 07:31:16 2025 -0400 release: fix docker image verification with telemetry disabled This commit fixes the verification of the FIPS docker image when telemetry is disabled. Epic: none Release note: none commit 8de4c80 Author: Alyshan Jahani <[email protected]> Date: Tue Aug 19 16:27:06 2025 -0400 contention: periodically log resolved contention events This commit adds logging to the flushAndResolve method in the contention event store. This method is called periodically at an interval defined by the (non-public) cluster setting sql.contention.event_store.resolution_interval. By logging aggregated contention event information, users now have the ability to find which waiting statements/txns are experiencing high contention through logs (previously this was only available via insights or SQL activity). Additionally, this aggregated event information contains the blocking txn and contended key - information that was previously only available via querying crdb_internal.transaction_contention_events. The logs are aggregated by a composite identifier of (waiting stmt, waiting txn, blocking txn, key), and the total contention time observed for that combination. They are structured logs. For example: ``` {"Timestamp":1756224167482848000,"EventType":"aggregated_contention_info", "WaitingStmtFingerprintId":"\\x000000000000007b", "WaitingTxnFingerprintId":"\\x00000000000001c8", "BlockingTxnFingerprintId":"\\x0000000000000315", "ContendedKey":"test-key-1","Duration":300000000} ``` Since this information is available via logs, it provides a solution for users who desire more persistence of contention event information. Part of: https://cockroachlabs.atlassian.net/browse/CRDB-53592 Release note (ops change): Adds logging of contention events.

aerfrei force-pushed the pts_protobuf branch from 562d4da to c090491 Compare July 7, 2025 20:26

aerfrei force-pushed the pts_protobuf branch 2 times, most recently from 5ec3b2c to 307fdc6 Compare July 18, 2025 20:02

aerfrei force-pushed the pts_protobuf branch 12 times, most recently from dbe2dc7 to 8012a08 Compare August 4, 2025 19:18

aerfrei changed the title ~~Pts protobuf~~ changefeedccl: protect each table with its own timestamp Aug 4, 2025

aerfrei requested review from KeithCh, andyyang890 and asg0451 August 4, 2025 19:21

aerfrei marked this pull request as ready for review August 4, 2025 19:21

aerfrei requested review from a team as code owners August 4, 2025 19:21

aerfrei requested review from jeffswenson and removed request for a team August 4, 2025 19:21

andyyang890 reviewed Aug 4, 2025

View reviewed changes

pkg/ccl/changefeedccl/changefeedbase/settings.go Outdated Show resolved Hide resolved

pkg/jobs/jobspb/jobs.proto Outdated Show resolved Hide resolved

msbutler requested a review from dt August 4, 2025 20:35

aerfrei force-pushed the pts_protobuf branch 2 times, most recently from 9f8dac7 to 5c07c88 Compare August 28, 2025 19:06

aerfrei requested a review from andyyang890 August 28, 2025 19:07

aerfrei force-pushed the pts_protobuf branch from 5c07c88 to 36c50ef Compare August 29, 2025 15:25

andyyang890 reviewed Aug 29, 2025

View reviewed changes

aerfrei force-pushed the pts_protobuf branch 5 times, most recently from 3dde139 to 76dae3e Compare August 29, 2025 18:08

aerfrei requested a review from andyyang890 August 29, 2025 20:02

andyyang890 reviewed Sep 2, 2025

View reviewed changes

aerfrei force-pushed the pts_protobuf branch from 76dae3e to 48dc12c Compare September 2, 2025 15:28

rafiss removed the request for review from a team September 2, 2025 15:30

aerfrei force-pushed the pts_protobuf branch from 48dc12c to 021bb15 Compare September 2, 2025 15:50

aerfrei requested a review from andyyang890 September 2, 2025 16:42

andyyang890 approved these changes Sep 2, 2025

View reviewed changes

craig bot merged commit c1e847b into cockroachdb:master Sep 2, 2025
23 checks passed

celeste-cockroachdb bot added the target-release-25.4.0 label Sep 2, 2025

andyyang890 mentioned this pull request Sep 17, 2025

changefeedccl: remove bucketing optimization for per-table PTS #153681

Closed

celeste-cockroachdb bot added v25.4.0-prerelease and removed target-release-25.4.0 labels Oct 1, 2025

Uh oh!

changefeedccl: protect lagging tables with its own timestamp #149616

changefeedccl: protect lagging tables with its own timestamp #149616

Uh oh!

Conversation

aerfrei commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cockroach-teamcity commented Jul 7, 2025

Uh oh!

blathers-crl bot commented Jul 18, 2025

Uh oh!

Uh oh!

Uh oh!

msbutler commented Aug 4, 2025

Uh oh!

andyyang890 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andyyang890 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andyyang890 left a comment

Choose a reason for hiding this comment

Uh oh!

aerfrei commented Sep 2, 2025

Uh oh!

craig bot commented Sep 2, 2025

Uh oh!

aerfrei commented Sep 2, 2025

Uh oh!

craig bot commented Sep 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

aerfrei commented Jul 7, 2025 •

edited

Loading