What's the best way of diagnosing scheduler memory issues?

[max-sixty]

I don't have a repro unfortunately, but my question is limited to how to investigate problems.

I have a job that frequently will cause the scheduler to start accumulating memory, eventually to 400GB of memory and then failing (the scheduler machine has that much memory).

I think I've read the docs thoroughly, but can't find any reason the scheduler should do this beyond having large values in tasks. I'm fairly confident that the client is not submitting any large values in tasks — it starts with bag.from_sequence(date_list) and has a series of .map calls.

Are there other reasons? How could I see what the scheduler is holding?

Thanks in advance!

[mrocklin]

Question: if you call client.restart() does the leak go away or not? client.restart() flushes out most of the scheduler state, but doesn't restart the process. This still won't give me personally enough information to tell you what's going on, but it may be somewhat informative.

…

On Tue, Jun 29, 2021 at 4:17 PM Maximilian Roos ***@***.***> wrote: I don't have a repro unfortunately, but my question is limited to how to investigate problems. I have a job that frequently will cause the scheduler to start accumulating memory, eventually to 400GB of memory and then failing (the scheduler machine has that much memory). I think I've read the docs thoroughly, but can't find any reason the scheduler should do this beyond having large values in tasks. I'm fairly confident that the client is not submitting any large values in tasks — it starts with bag.from_sequence(date_list) and has a series of .map calls. Are there other reasons? How could I see what the scheduler is holding? Thanks in advance! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4998>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAFTBFCGL6XMMWXOM3TVJIBZANCNFSM47RFYFJQ> .

[max-sixty]

On client.restart(), the memory doesn't get cleaned up. It's now running at 10% CPU with 80GB in memory (I cut it early this time). When I terminate it, it states full garbage collection released 9.38 GiB from 46012 reference cycles. Though that's not that big a share of the total heap.

(And tbc, I recognize that without a repro it's very difficult to help, so I really was mostly looking for how I could investigate it on my own — I'm not even sure how the data gets there, let alone stays there)

[TomAugspurger]

You might try https://pympler.readthedocs.io/en/latest/muppy.html#muppy, specifically https://pympler.readthedocs.io/en/latest/muppy.html#the-tracker-module. I think that uses sys.getsizeof, so it relies on objects correctly reporting their size.

https://gist.github.com/TomAugspurger/f870d87456695cf47bafd5a8a5ae450e is a (not working) work-in-progress WorkerPlugin for periodically checking what objects are alive in on the workers. You could perhaps adapt that for a SchedulerPlugin.

[max-sixty]

Thanks @TomAugspurger . I had been hoping there might be an existing approach for seeing which objects the scheduler was sent, to try and evaluate why it was building up such a memory pressure. I'll see what I can do in the meantime. Cheers!

[max-sixty]

I'll close this. FWIW running without fusing somehow seems to have reduced the scheduler memory load to peaking at ~20GB. If I can pull a reproduction out of this I'll post that (but unfortunately I think the likelihood is fairly low)