Dynamic chunking doc update and improvements #395
Description
from a @toliwaga, @stefancoe conversation:
Dynamic chunking will do its best to adjust chunks to hit the specified chunk size, not the available memory. So if the chunk size setting is too high, it will do its best adjust the number of rows per chunk to achieve that mark, even if there isn’t enough RAM to accommodate it.
So, for now, you do still have to set a guestimated chunk_size and adjust it upwards or downwards according to the observed memory footprint activitysim leaves relative to the available resources.
As the discussion today indicated, you can imagine also allowing activitysim to decide how many sub-processes to deploy – analogously to RAM.
Getting such a ready-to-wear off the-rack capability is a bit tricky because we can’t necessarily assume that it is ok to just gobble up all available machine resources, but it would be really handy if you could just say “use 85% of the total CPU and 80% of the processors” and let it roll…