Parts of work redone when a dask worker dies #1154
Description
This arose in the context of sgkit-dev/sgkit-publication#35 (comment) where dask workers being rotated due to slow memory leaks caused work to be redone and the VCF parse to never complete. This likely also happens with other format parsers.
Upon investigation, it was found that dask requires the result of each parse task to be in cluster memory, despite this just being a None. Possible ways to fix this:
1: Make a note on disk that a chunk has completed being parsed as make the second parse task a no-op. Downside of this is lots of files on disk.
2: Cancel the dask future after completion. Downside of this is a confusing dask progress bar that slowly gets less work to be done without any looking completed.
3: Some tweak to the dask memory manager to replicate parse task results
2 seems to be the best option.