-
Notifications
You must be signed in to change notification settings - Fork 740
schemeshard: stats processing optimizations #27165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
schemeshard: stats processing optimizations #27165
Conversation
|
🟢 |
47f60fd to
7e4fd77
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
7e4fd77 to
aff4164
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
PathShardsLimit() performs pedantic validation of the path's current shards count by recalculating it through iteration over ShardInfos of the entire database and matching them to the path (!). See CollectAllShards(). That is unreasonable and too slow for the hot spot that is PersistSingleStats(). (Though appropriate for slow paths like TSubOperation::Propose() etc.) Replacing PathShardsLimit() also eliminates aditional TPath construction and hashmap lookup.
aff4164 to
1ce4a58
Compare
|
⚪
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
|
⚪ ⚪ Ya make output | Test bloat | Test bloat
⚪ Ya make output | Test bloat | Test bloat | Test bloat
🟢
*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation |
Cherry-pick from `main`: - 626410b, #27165 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 - fe49740, #28834 - 3950ae8, #29664 - 585a708, #29813 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 - fe49740, #28834 - 3950ae8, #29664 - 585a708, #29813 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 - fe49740, #28834 - 3950ae8, #29664 - 585a708, #29813 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfoscommits/6fefb4b0d08afdd7a9430d54d3d5d70e7bf00100) - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 - fe49740, #28834 - 3950ae8, #29664 - 585a708, #29813 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Cherry-pick from `main`: - 626410b, #27165 - fe49740, #28834 - 3950ae8, #29664 - 585a708, #29813 Streamline and optimize datashard statistics processing. Profile guided optimizations of PersistSingleStats() (in synthetic test). Total gain is around 30%. - remove unreasonable iteration over entire ShardInfos - single Now() timestamp for entire stats batch - optimize number of lookups - stop building now unnecessary storage pool kind mappings - remove table/store aggregated stats copying - collect ExternalBlobsEnabled only on PartitionConfig change - replace ETxType->CounterId map with absl::flat_hash_map - remove extra OpType->TxType lookup - remove call to GetMainTableForIndex for not-index-table shards
Streamline and optimize datashard statistics processing.
Changes are mostly independent. Better reviewed commit-by-commit.
Profile guided optimizations of PersistSingleStats() (in synthetic test).
Total gain is around 30%.