[RFC] Simplify accelerator API, add training type argument

[SeanNaren]

🚀 Feature

After the accelerator refactor, we've added some nicety around the TrainingTypePlugin, which now represents all logic that sits on top of an accelerator. However our API seems to be based on the old paradigm where the accelerator/distributed backend were tied, and has bit us already #6089

What I suggest is we do the following:

# Correct
Trainer(gpus=1, accelerator='ddp')  # this is fine. should be set to ddp
Trainer(gpus=1, plugins='ddp_sharded', accelerator='gpu')
Trainer(gpus=1, plugins='ddp_sharded')  # implicit. assume GPU for ddp_sharded as it is the only supported accelerator

# Incorrect
Trainer(gpus=1, accelerator='ddp_sharded')  # throw misconfiguration exception
Trainer(gpus=1, plugins='ddp_sharded', accelerator='ddp') # this should throw a deprecation warning

# The Future

# Plugins can be TrainingTypePlugins, PrecisionPlugins, or whatever the user has provided.
# For precision, we don't do `Trainer(plugins='16bit_precision')` but instead `Trainer(precision=16)` 
# so why do we use the plugins flag for the training type?

Trainer(training_type='ddp', accelerator='gpu')
Trainer(training_type='ddp_spawn', accelerator='cpu')

# so Trainer(plugins=...) is left for cluster_environments, SLURM, ...

Incompatible Plugin/Accelerator

This is something I haven't hashed out but let's say the user does:

Trainer(training_type='ddp_sharded', accelerator='tpu')

This currently is not compatible. We should throw an exception of some sort, but this delves into whitelisting/blacklisting support plugins which could get a bit unwieldy. An alternative is to just assume that the user knows enough about compatibility already.

Backwards compatibility

We allow the TrainingTypePlugin to be specified via the accelerator trainer flag but throw a warning that this will be deprecated in the future.

cc @ananthsub @carmocca @awaelchli @justusschock @tchaton @Borda @kaushikb11 @williamFalcon

[Borda]

just a quick remark, is this a typo?

Trainer(gpus=1, training_type='ddp_spawn', accelerator='cpu')

[carmocca]

These are all the possibilities we currently have

training_types = [
    "single",
    "dp",
    "ddp",
    "ddp2",
    "ddp_spawn",
    "ddp_sharded",
    "ddp_sharded_spawn",
    "horovod",
]
accelerators = ["cpu", "gpu", "tpu"]

import itertools
list(itertools.product(accelerators, training_types))
# comment indicates how it is currently set
[
    ✅ ("cpu", "single"),  # gpus=0
    👎 ("cpu", "ddp"),
    👎 ("cpu", "ddp2"),
    ✅ ("cpu", "ddp_spawn"),  # accelerator=ddp_cpu, num_processes=n
    👎 ("cpu", "ddp_sharded"),
    👎 ("cpu", "ddp_sharded_spawn"),
    👎 ("cpu", "dp"),
    ✅ ("cpu", "horovod"),  # accelerator=horovod

    ✅ ("gpu", "single"),  # gpus=n
    ✅ ("gpu", "ddp"),  # accelerator=ddp, gpus=n
    ✅ ("gpu", "ddp2"),  # accelerator=ddp2, gpus=n
    ✅ ("gpu", "ddp_spawn"),  # accelerator=ddp_spawn, gpus=n
    ✅ ("gpu", "ddp_sharded"),  #  accelerator=ddp, gpus=n, plugins=ddp_sharded
    ✅ ("gpu", "ddp_sharded_spawn"),  #  accelerator=ddp, gpus=n, plugins=ddp_sharded_spawn
    ✅ ("gpu", "dp"),  # accelerator=dp, gpus=n
    ✅ ("gpu", "horovod"),  # accelerator=horovod, gpus=n

    ✅ ("tpu", "single"),  # tpu_cores=n
    👎 ("tpu", "ddp"),
    👎 ("tpu", "ddp2"),
    ✅ ("tpu", "ddp_spawn"),  # accelerator=ddp_spawn, tpu_cores=n
    👎 ("tpu", "ddp_sharded"),
    👎 ("tpu", "ddp_sharded_spawn"),
    👎 ("tpu", "dp"),
    👎 ("tpu", "horovod"),
]

(Might have made a mistake on some, feel free to edit and correct)

DeepSpeed, RPC, RPC sequential are just plugins

We should throw an exception of some sort, but this delves into whitelisting/blacklisting support plugins which could get a bit unwieldy

I don't think we have a choice. I wouldn't leave it to the user to understand how each of these combinations works.
Also no user will be implementing their own training_type or accelerator so it should be safe for us to decide which of these are allowed.

Also, we might want to evaluate standardizing how the number of devices is set. We currently have num_processes, gpus, and tpu_cores. They could be merged into one devices or num_processes

[SeanNaren]

Thanks @carmocca!

Just to play devils advocate, we've gotten this far without having to worry about blacklisting of sorts. Assuming the user knows what they are doing isn't as fatal as we're making it out to believe, as most of these flags requires checking docs to see what is supported. Regardless on the decision of blacklisting/whitelisting I think this should remain a separate endeavour to the API but it is an important extension that the API should support!

I agree on standardising the devices! Definitely a separate case that can extend from this API