[SPARK-43364][SS][DOCS] Add docs for RocksDB state store memory management #41042
Conversation
|
@HeartSaVioR @siying - please take a look. Thanks |
anishshri-db
changed the title
| <tr> | ||
| <td>spark.sql.streaming.stateStore.rocksdb.highPriorityPoolRatio</td> | ||
| <td>Total memory to be occupied by filter and index blocks as a fraction of memory allocated across all RocksDB instances on a single node.</td> | ||
| <td>0.1</td> |
There was a problem hiding this comment.
I didn't realize the default is 0.1. If we don't have a special study, 0.5 is a good value to start with.
There was a problem hiding this comment.
Yes, this is for the high pri pool blocks. The default for write buffer ratio is 0.5. Do we not need to keep some fraction for elements in the regular block cache too ?
| @@ -2360,8 +2360,35 @@ Here are the configs regarding to RocksDB instance of the state store provider: | |||
| <td>The maximum number of MemTables in RocksDB, both active and immutable. Value of -1 means that RocksDB internal default values will be used</td> | |||
| <td>-1</td> | |||
| </tr> | |||
| <tr> | |||
| <td>spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage</td> | |||
There was a problem hiding this comment.
Wonder if we can add them into SQLConf or StaticSQLConf so the docs can be generated, and to make it clear which value is the default.
For instance, we automatically generate the docs from SQL conf.
There was a problem hiding this comment.
Can be done separately in another PR.
There was a problem hiding this comment.
Maybe better to defer this till we make RocksDB state store provider as default implementation.
|
cc @HeartSaVioR and @xuanyuanking FYI |
|
Thanks! Merging to master. |
HeartSaVioR
changed the title
…ement ### What changes were proposed in this pull request? Add docs for RocksDB state store memory management ### Why are the changes needed? Docs only change ### Does this PR introduce _any_ user-facing change? N/A ### How was this patch tested? N/A Closes apache#41042 from anishshri-db/task/SPARK-43364. Authored-by: Anish Shrigondekar <[email protected]> Signed-off-by: Jungtaek Lim <[email protected]>
What changes were proposed in this pull request?
Add docs for RocksDB state store memory management
Why are the changes needed?
Docs only change
Does this PR introduce any user-facing change?
N/A
How was this patch tested?
N/A