[RFC] Future of `gpus/ipus/tpu_cores` with respect to `devices`

[SeanNaren]

Proposed refactoring or deprecation

Currently we have two methods to specifying devices. Let's take GPUs for example:

1. The standard case that we've all grown used to and are mostly aware of.

trainer = Trainer(gpus=2)

2. Introduced in 1.5, tries to make the number of devices agnostic. This means if you specify accelerator='tpu' we automatically know to use 2 TPU cores.

trainer = Trainer(devices=2, accelerator='gpu')

Recently, it has come up in #10404 (comment) that we may want to deprecate and prevent further device specific names from appearing in the Trainer (such as hpus).

Related conversation #9053 (comment)

I see two options:

🚀 We keep both device specific arguments (gpus tpu_cores ipus for the Trainer) and devices
👀 We drop gpus tpu_cores ipus in the future and fully rely on devices. (Potentially this would likely be done in Lightning 2.0, instead of after 2 minor releases)

cc @kaushikb11 @justusschock @ananthsub @awaelchli

[ananthsub]

IMO we should follow the contributing guidelines: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md#main-core-value-one-less-thing-to-remember

Having multiple options in the public API to do the same thing is really confusing.

I'm in favor of devices=X, accelerator=Y since it's clearer how extensible this can be.

[four4fish]

+1, totally agree

Current device related flags are confusing. Multiple flags partially overlap and interfere each other. When multiple flags passed in, we define prioritize and ignore some of the flags.

For example:
gpu=2, device=3 device will be ignored.
gpu=2, cpu=2, accelerator=cpu what will happen? I think cpu with num_process=2?

I prefer option 2, drop gpus, tpu_cores, ipus in the future and fully rely on devices
And can we have devices be int, not set =auto?

With this option: accelerator flag for device_type, devices (probably rename to devices_num?) for device_number. It's also scalable for new device types like hpus

[williamFalcon]

I think going from this:

Trainer(gpus=2)

to

Trainer(devices=2, accelerator='gpu')

is a major step backwards in usability. now users have to dig into docs to understand how to use things. it definitely violates the "one-less-thing-to-remember" part of the API.

I guess, I'm just wondering why we're exploring this? I thought we were already pretty stable on the device API stuff

[justusschock]

@williamFalcon The more kinds of accelerators we get, the more flags we will also have. Switching from Trainer(gpus=8) to Trainer(tpu_cores=8) also requires users to dig through the docs. Actually I find it easier to have Trainer(devices=2, accelerator='gpu'/'tpu') as the flags stay the same, it is easier to remember and also scaling better. So personally this would be the "one-less-thing-to-remember" for me.

Also I suspect, we would likely have the accelerator defaulting to 'auto' then which means that Trainer(devices=8) would run on gpu if available, on tpu if available and if no special accelerator is available it would fall back to cpu.

[tchaton]

@williamFalcon As @justusschock shared, the previous approach doesn't scale well and makes discoverability harder.

Furthermore, the new API provides an auto as follows:

Trainer(devices="auto", accelerator="auto")

which would make the code runnable on every hardware without any code changes. Which isn't possible with the previous API.

And we could even support num_nodes discovery too.

Trainer(devices="auto", accelerator="auto", num_nodes="auto")

[SeanNaren]

To address the discoverability issue, isn't it common to import the Trainer and see what parameters are available? Isn't this more common that going to the docs to find the parameter?

I opened the issue as I felt it was important as a community we come to an agreement as the idea was floating around a few PRs (with inconsistent agreement). It's important to have one single direction here (especially as we introduce other accelerators). I do strongly disagree with removing gpus/tpu_cores/ipus/hpus/cpus from the Trainer for primarily ease/discoverability.

I think it would be beneficial to try get community votes on this, so maybe a post on our General slack channel is warranted?

[ananthsub]

Even something like gpus as lightning defines them today is ambiguous. PyTorch also supports AMD GPUs: https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/

But this isn't supported at all with Lightning because when specifying gpus on the Trainer constructor, the Trainer assumes NVidia & CUDA are being used. On the other hand, PyTorch's device allows for more backends.

In my head the accelerator in Lightning maps to the torch device being used. By using the same sematics PyTorch offers, Lightning can keep parity more easily and smoothen the transition for users coming from vanilla PyTorch

[tchaton]

Adding to the conversation and @ananthsub comment. In this issue, a user is requesting to adding MIG for A100-like a machine: #10529.

This is another example of arguments like gpus, tpu-cores can grow out of control, and the need for a single device / accelerator arguments.

[daniellepintz]

@tchaton do we have a consensus to move forward with this issue?

[zippeurfou]

Maybe thought that is a bit different there.
Going back to @williamFalcon argument that:
Trainer(gpus=2) to Trainer(devices=2, accelerator='gpu') is more work for the user.
My question here is that as far as I understand a correct me if I am wrong you have CPU and then either gpu/tpu/hpu/...
That is why you can have something as follow:
Trainer(devices=2, accelerator='gpu') where accelerator is only one combination.
I also assume that our users "don't" want to care if it is gpu/tpu/hpu.. as their code will remain the same as long as it is not CPU and even if it is CPU PL help you there to make it seamless.
Finally, we can automatically detect what kind of accelerator you have available today with auto.
That being said what if we had a "wrapper" around anything that is non CPU such that we can keep the same structure while making it "easy" for the users.
ie. Trainer(cpus=2,xpus=2) this will automatically find if x is gpu/tpu/hpu.
Then we allow default to be auto ie. Trainer(cpus=null,xpu=null) or could use -1 for example.