Skip to content

Conversation

@ijon
Copy link
Collaborator

@ijon ijon commented Dec 1, 2025

Cherry-pick from main:

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

  • remove unreasonable iteration over entire ShardInfos
  • single Now() timestamp for entire stats batch
  • optimize number of lookups
  • stop building now unnecessary storage pool kind mappings
  • remove table/store aggregated stats copying
  • collect ExternalBlobsEnabled only on PartitionConfig change
  • replace ETxType->CounterId map with absl::flat_hash_map
  • remove extra OpType->TxType lookup
  • remove call to GetMainTableForIndex for not-index-table shards

ijon added 4 commits December 1, 2025 18:34
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
Protect shard statistics processing from missing channel binding information on table shards for old clusters or old databases still served by the root schemeshard.
Fix finding storage pool kind for a tablet's channel for tables created before 25-3. TChannelBindings persisted at schemeshard at earlier versions don't have StoragePoolKind field filled.
Fix disk space aggregation for column tables.
Make subdomain and storage pool kind aggregation levels separation more robust.
Add tests on subdomain level aggregation.
@ijon ijon changed the title schemeshard: stats processing optimizations (#27165) 25-3: schemeshard: stats processing opt 1 Dec 1, 2025
@ijon ijon changed the title 25-3: schemeshard: stats processing opt 1 25-3: schemeshard: stats processing opt 1b Dec 1, 2025
@github-actions
Copy link

github-actions bot commented Dec 1, 2025

2025-12-01 15:39:22 UTC Pre-commit check linux-x86_64-release-asan for d44a576 has started.
2025-12-01 15:40:01 UTC Artifacts will be uploaded here
2025-12-01 15:44:21 UTC ya make is running...
🟡 2025-12-01 18:00:11 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15937 15277 0 321 323 16

🟢 2025-12-01 18:01:32 UTC Build successful.
🟢 2025-12-01 18:01:57 UTC ydbd size 3.8 GiB changed* by +15.5 KiB, which is < 100.0 KiB vs stable-25-3: OK

ydbd size dash stable-25-3: eb91e15 merge: d44a576 diff diff %
ydbd size 4 065 887 008 Bytes 4 065 902 832 Bytes +15.5 KiB +0.000%
ydbd stripped size 1 514 804 224 Bytes 1 514 817 600 Bytes +13.1 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

🟢 2025-12-01 15:40:10 UTC The validation of the Pull Request description is successful.

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

2025-12-01 15:39:26 UTC Pre-commit check linux-x86_64-relwithdebinfo for d44a576 has started.
2025-12-01 15:39:39 UTC Artifacts will be uploaded here
2025-12-01 15:43:16 UTC ya make is running...
🟡 2025-12-01 17:16:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
38812 36134 0 7 2650 21

2025-12-01 17:19:12 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-12-01 17:35:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
566 (only retried tests) 550 0 1 0 15

2025-12-01 17:36:00 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-12-01 17:55:06 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
552 (only retried tests) 537 0 1 0 14

🟢 2025-12-01 17:55:14 UTC Build successful.
🟢 2025-12-01 17:55:36 UTC ydbd size 2.3 GiB changed* by +20.9 KiB, which is < 100.0 KiB vs stable-25-3: OK

ydbd size dash stable-25-3: eb91e15 merge: d44a576 diff diff %
ydbd size 2 442 813 944 Bytes 2 442 835 376 Bytes +20.9 KiB +0.001%
ydbd stripped size 522 835 976 Bytes 522 847 688 Bytes +11.4 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ijon ijon marked this pull request as ready for review December 2, 2025 09:10
@ijon ijon requested a review from a team as a code owner December 2, 2025 09:10
@ijon ijon enabled auto-merge December 2, 2025 10:36
@maximyurchuk maximyurchuk disabled auto-merge December 2, 2025 12:04
@maximyurchuk maximyurchuk merged commit 792de81 into ydb-platform:stable-25-3 Dec 2, 2025
11 of 14 checks passed
@ijon ijon deleted the merge/25-3/schemeshard-stats-processing-opt-1-fixed branch December 2, 2025 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants