Skip to content

Conversation

@ijon
Copy link
Collaborator

@ijon ijon commented Nov 29, 2025

Fix-up to:

Fix disk space aggregation for column tables.
Separate subdomain and storage pool kind aggregation level more robust.
Add tests on subdomain level aggregation.

Copilot AI review requested due to automatic review settings November 29, 2025 19:20
@ijon ijon requested review from a team as code owners November 29, 2025 19:20
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes disk space aggregation for column tables by refactoring how disk space usage deltas are processed. The key issue addressed is that column table statistics were not being properly aggregated at the subdomain level after a schemeshard reboot.

Key changes:

  • Removed unused diskSpaceUsageDelta parameter from UpdateTableStats methods for column tables
  • Changed AggrDiskSpaceUsage to treat disk space usage delta as an ordered vector instead of a map, with total space at index 0 and per-pool deltas at subsequent indices
  • Added comprehensive test coverage for disk space usage with various table configurations (regular tables, column tables in stores, standalone column tables)

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
ydb/core/tx/schemeshard/ut_subdomain/ya.make Added dependencies for columnshard testing utilities
ydb/core/tx/schemeshard/ut_subdomain/ut_subdomain.cpp Added extensive test coverage for disk space usage tracking across different table types and configurations
ydb/core/tx/schemeshard/schemeshard_info_types.h Removed diskSpaceUsageDelta parameter from UpdateTableStats signature
ydb/core/tx/schemeshard/schemeshard_info_types.cpp Refactored AggrDiskSpaceUsage to treat delta as ordered vector; simplified UpdateTableStats implementation
ydb/core/tx/schemeshard/schemeshard__table_stats.cpp Updated call site to match new UpdateTableStats signature
ydb/core/tx/schemeshard/olap/table/table.h Updated UpdateTableStats signature and implementation for column tables

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

@github-actions
Copy link

github-actions bot commented Nov 29, 2025

🟢 2025-12-01 08:02:13 UTC The validation of the Pull Request description is successful.

@ijon ijon force-pushed the schemeshard-stats-processing-opt-fix2 branch from cd0003b to 16f719b Compare November 29, 2025 19:37
@github-actions
Copy link

github-actions bot commented Nov 29, 2025

2025-11-29 19:41:07 UTC Pre-commit check linux-x86_64-release-asan for 00ef793 has started.
2025-11-29 19:41:26 UTC Artifacts will be uploaded here
2025-11-29 19:43:37 UTC ya make is running...
🟡 2025-11-29 21:02:33 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
11962 11807 0 125 18 12

🟢 2025-11-29 21:02:43 UTC Build successful.
🟢 2025-11-29 21:03:09 UTC ydbd size 3.8 GiB changed* by +2.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4d52fb5 merge: 00ef793 diff diff %
ydbd size 4 120 837 760 Bytes 4 120 840 336 Bytes +2.5 KiB +0.000%
ydbd stripped size 1 530 278 072 Bytes 1 530 278 584 Bytes +512 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Nov 29, 2025

2025-11-29 19:41:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for 00ef793 has started.
2025-11-29 19:41:34 UTC Artifacts will be uploaded here
2025-11-29 19:43:46 UTC ya make is running...
🟡 2025-11-29 21:20:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39592 36724 0 1 2842 25

2025-11-29 21:20:57 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-11-29 21:31:19 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
44 (only retried tests) 31 0 0 0 13

🟢 2025-11-29 21:31:26 UTC Build successful.
🟢 2025-11-29 21:31:48 UTC ydbd size 2.3 GiB changed* by +1.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 4d52fb5 merge: 00ef793 diff diff %
ydbd size 2 462 318 488 Bytes 2 462 320 160 Bytes +1.6 KiB +0.000%
ydbd stripped size 524 149 216 Bytes 524 149 344 Bytes +128 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ijon ijon linked an issue Nov 29, 2025 that may be closed by this pull request
@ijon ijon merged commit 585a708 into ydb-platform:main Dec 1, 2025
11 checks passed
@ijon ijon deleted the schemeshard-stats-processing-opt-fix2 branch December 1, 2025 09:22
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
- fix disk space aggregation for column tables
- make subdomain and storage pool kind aggregation levels separation more robust
- add tests on subdomain level aggregation
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

schemeshard: optimize statistics processing

3 participants