Skip to content

Conversation

@ijon
Copy link
Collaborator

@ijon ijon commented Oct 20, 2025

@github-actions
Copy link

github-actions bot commented Oct 20, 2025

🟢 2025-10-23 19:57:54 UTC The validation of the Pull Request description is successful.

@ijon ijon force-pushed the schemeshard-stats-processing-opt branch from 47f60fd to 7e4fd77 Compare October 20, 2025 16:01
@ydb-platform ydb-platform deleted a comment from github-actions bot Oct 20, 2025
@ydb-platform ydb-platform deleted a comment from github-actions bot Oct 20, 2025
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@ijon ijon force-pushed the schemeshard-stats-processing-opt branch from 7e4fd77 to aff4164 Compare October 22, 2025 22:04
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

ijon added 9 commits October 23, 2025 16:18
PathShardsLimit() performs pedantic validation of the path's current shards
count by recalculating it through iteration over ShardInfos of the entire
database and matching them to the path (!). See CollectAllShards().
That is unreasonable and too slow for the hot spot that is PersistSingleStats().
(Though appropriate for slow paths like TSubOperation::Propose() etc.)

Replacing PathShardsLimit() also eliminates aditional TPath construction
and hashmap lookup.
@ijon ijon force-pushed the schemeshard-stats-processing-opt branch from aff4164 to 1ce4a58 Compare October 23, 2025 13:19
@github-actions
Copy link

github-actions bot commented Oct 23, 2025

2025-10-23 13:20:42 UTC Pre-commit check linux-x86_64-release-asan for 69bc8af has started.
2025-10-23 13:20:57 UTC Artifacts will be uploaded here
2025-10-23 13:24:34 UTC ya make is running...
🟡 2025-10-23 15:33:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15958 15431 0 254 258 15

🟢 2025-10-23 15:33:07 UTC Build successful.
🟢 2025-10-23 15:33:28 UTC ydbd size 3.8 GiB changed* by +28.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 74cf4c1 merge: 69bc8af diff diff %
ydbd size 4 053 820 208 Bytes 4 053 849 424 Bytes +28.5 KiB +0.001%
ydbd stripped size 1 505 358 464 Bytes 1 505 382 464 Bytes +23.4 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Oct 23, 2025

2025-10-23 13:20:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for 69bc8af has started.
2025-10-23 13:20:57 UTC Artifacts will be uploaded here
2025-10-23 13:24:35 UTC ya make is running...
🟡 2025-10-23 15:00:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39215 36429 0 6 2756 24

2025-10-23 15:00:21 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-10-23 15:20:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
8106 (only retried tests) 8081 0 1 0 24

2025-10-23 15:20:43 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-10-23 15:39:32 UTC Tests successful.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
439 (only retried tests) 420 0 0 0 19

🟢 2025-10-23 15:39:35 UTC Build successful.
🟢 2025-10-23 15:39:53 UTC ydbd size 2.3 GiB changed* by +22.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 74cf4c1 merge: 69bc8af diff diff %
ydbd size 2 436 392 128 Bytes 2 436 415 504 Bytes +22.8 KiB +0.001%
ydbd stripped size 517 948 936 Bytes 517 961 480 Bytes +12.2 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

ijon added a commit that referenced this pull request Nov 12, 2025
Cherry-pick from `main`:
- 626410b, #27165

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Nov 12, 2025
Cherry-pick from `main`:
- 626410b, #27165

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Nov 12, 2025
Cherry-pick from `main`:
- 626410b, #27165

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Nov 12, 2025
Cherry-pick from `main`:
- 626410b, #27165

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
qyryq pushed a commit to qyryq/ydb that referenced this pull request Nov 24, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit to ijon/ydb that referenced this pull request Dec 1, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100)
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
ijon added a commit that referenced this pull request Dec 2, 2025
Cherry-pick from `main`:
- 626410b, #27165
- fe49740, #28834
- 3950ae8, #29664
- 585a708, #29813

Streamline and optimize datashard statistics processing.

Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%.

- remove unreasonable iteration over entire ShardInfos
- single Now() timestamp for entire stats batch
- optimize number of lookups
- stop building now unnecessary storage pool kind mappings
- remove table/store aggregated stats copying
- collect ExternalBlobsEnabled only on PartitionConfig change
- replace ETxType->CounterId map with absl::flat_hash_map
- remove extra OpType->TxType lookup
- remove call to GetMainTableForIndex for not-index-table shards
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

schemeshard: optimize statistics processing

3 participants