Provide parameter for "dask-worker"

[mrocklin]

Issue rapidsai/dask-cuda#3 proposes creating a dask-cuda-worker alternative to dask-worker which starts one process per GPU with appropriate environment variables. If we go ahead with that then it would be convenient to compose that approach with dask-jobqueue. This would be particularly easy if the following line could be made configurable to allow other executables

dask-jobqueue/dask_jobqueue/core.py

Line 254 in 5a5585d

dask_worker_command = '%(python)s -m distributed.cli.dask_worker' % dict(python=python)

Thoughts?

[lesteve]

This looks fine to me, maybe via an additional dask_worker_module argument or something similar?

Am I right in thinking that using dask-cuda-worker mostly makes sense when your job books at least two GPUs?

Also are all the command-line arguments of dask-worker applicable to dask-cuda-worker?

I am trying to wrap my head around what the difference would be in the setup I am familiar with, i.e. CUDA_VISIBLE_DEVICES is set by SGE ... I have to say that the main use case people around me have is embarassingly parallel workflow where each job only use one GPU so maybe not the target audience for dask-cuda-worker.

[jhamman]

@mrocklin - no objections from me. I can see this being useful for a few other applications as well.

[mrocklin]

Am I right in thinking that using dask-cuda-worker mostly makes sense when your job books at least two GPUs?

Yes

Also are all the command-line arguments of dask-worker applicable to dask-cuda-worker?

dask-cuda-worker doesn't exist currently, but yes, I would expect it to be a pretty lightweight wrapper around dask-worker

i.e. CUDA_VISIBLE_DEVICES is set by SGE

That would also work well. I'd be curious to know how you do this generally. Do you submit multiple jobs to the same node and have them use different values for CUDA_VISIBLE_DEVICES? If so, how do you organize this?

[lesteve]

That would also work well. I'd be curious to know how you do this generally. Do you submit multiple jobs to the same node and have them use different values for CUDA_VISIBLE_DEVICES? If so, how do you organize this?

Here a snippet to illustrate the typical dask usage with multiple GPU in my team:

# 'some_gpu_queue' is a queue where all the nodes have the same number of GPUs (typically 4)
cluster = SGECluster(queue='some_gpu_queue', cores=1, processes=1, ...)
client = Client(cluster)

# 10 workers with one GPU per worker
# generally workers are on different nodes but some workers may be on the same node
cluster.scale(10)

client.map(function_that_uses_a_gpu, list_of_inputs)

In other words, they submit jobs with one GPU per job and then have an embarrassingly parallel workflow (e.g. grid-search), so they don't really care so much whether jobs end up on the same node or not.
Probably two reasons for this:

• I don't think they do a lot of multi-GPU (in the sense of multiple GPU on the same node where you need to communicate a lot between GPUs)
• GPU is a scarce resource, if you insist you want two GPUs on the same node you may wait longer in the queue

From a cluster user perspective you don't have to take care of CUDA_VISIBLE_DEVICES. If you really want to know more: we have a hack (based on SGE parallel environments and a lock file per GPU): when the job starts on a node, it finds an unused GPU (looking at lock files) and sets CUDA_VISIBLE_DEVICES.

@guillaumeeb may have good insights about how they use SLURM with GPUs. I don't know much about SLURM but reading their doc ("GPU management" section) it looks like CUDA_VISIBLE_DEVICES is set by the queuing system when the job starts so this seems quite similar to our SGE setup.

Another approach I have seen on another cluster at Inria (I don't think anyone uses dask on this cluster): they use the OAR scheduler which I think uses cgroups to isolate resources within a job. Say you require 2 GPUs on a node that has 8 GPUs. Inside your job everything looks as if you only have 2 GPUs available (e.g. nvidia-smi lists only 2 GPUs).

[mrocklin]

when the job starts on a node, it finds an unused GPU (looking at lock files) and sets CUDA_VISIBLE_DEVICES

That sounds like a nice solution. Do we know if approaches like this are standard?

[guillaumeeb]

We're using PBSPro on our cluster. Currently we have only a few GPUs, and we are looking at ways to allow the selection of only one of them per node (which we obviously don't already support).

From what I've seen so far, solutions are based on setting CUDA_VISIBLE_DEVICES in a way close to what @lesteve is mentionning for SGE or Slurm. So I would assume this is a pretty standard solution with queuing systems.

Getting back to the initial question, and based on the following discussion, you could probably use dask-cuda without a specific dask-cuda-worker in dask-jobqueue. Suppose you have 4 GPUs nodes with 32 cores:

cluster = PBSCluster(cores=8, resource_spec='select=1:ncpus=8:ngpus=1:mem=64GB')

Then you use the queuing system to start one job and worker process per GPUs. But obviously, you probably won't be able to do, or can you?

cluster = PBSCluster(cores=32, processes=4, resource_spec='select=1:ncpus=32:ngpus=4:mem=256GB')

If I understand properly, this is where you could use:

cluster = PBSCluster(cores=32, queue=gpu, worker_cmd='dask-cuda-worker')

So in the end, no objection from me!