Distributed Traning

[petrex]

The goal is to evaluate the high-level approach to enable Pytorch/caffe2 distributed training

[iotamudelta]

redundant w/ #31 ?

[Delaunay]

I was able to use the distributed_parallel pytorch API. I modified a bit the micro_benchmark script you provide to check (modified version is there).

On a single node it scales well with the number of GPUs.
I did not have any issue with it, called the script the exact same way as I did for NVIDIA.
It was seamless

$ python launch.py --devices 0,1,2,3,4,5,6 micro_benchmarking_pytorch.py --batch-size 128 --network resnet101 --fp16 1
$ python launch.py --devices 0,1,2,3,4,5,6 micro_benchmarking_pytorch.py --batch-size 128 --network resnet101 --fp16 1

	fp16	fp32
GPU	AMD	AMD
1	225.82	155.54
2	442.4	301.73
3	645.38	447.69
4	844.21	590.78
5	1063.32	743.2
6	1262.94	887.77
7	1469.61	1019.73
8	1672.97	1186.46

[iotamudelta]

Closing as distributed training is integrated in both PyTorch and Caffe2 backends for some time w/o problems.