benchmark

Benchmark Suite

benchmark_plan.py

Benchmark a Transfer Plan

Prequisits

Compile the project using the provided Makefile by simply calling git submodule update --init && make in the base directory.

You will need:

- CUDA >= 9.2
- GNU g++ >= 6.3.0 and compatible with your CUDA version
- Python >= 3.0 with
    - Matplotlib
    - NumPy

Usage

usage: benchmark_plan.py [-h] [--dir DIR] [--repeats REPEATS]
                         [--maxsize MAXSIZE] [--minsize MINSIZE]
                         type plan

positional arguments:
  type               collective type
                     (all2all|all2all_async|scatter_gather|broadcast)
  plan               JSON which specifies the communication strategy

optional arguments:
  -h, --help         show this help message and exit
  --dir DIR          output directory (default: .)
  --repeats REPEATS  number of repeated executions (default: 5)
  --maxsize MAXSIZE  maximum overall amount of data to be communicated (bytes
                     log2) (default: 28)
  --minsize MINSIZE  minimum overall amount of data to be communicated (bytes
                     log2) (default: 12)

Note: Only supports all2all and all2all_async at the moment. Will be fixed soon...

Result

A csv file with the following format:

<input size #1 in bytes>,<runtime #1 in ms>,<runtime #2 in ms>,...
<input size #2 in bytes>,<runtime #1 in ms>,<runtime #2 in ms>,...
<input size #3 in bytes>,<runtime #1 in ms>,<runtime #2 in ms>,...
...

---

benchmark_all2all.py

Benchmark and Plot All2All Performance on DGX-1 or DGX-2 Topologies

Prequisits

Compile the project using the provided Makefile by simply calling make in the base directory.

Usage

This script runs a complete benchmark of all relevant all2all configurations on either DGX-1 or DGX-2 and automatically generates the corresponding visualizations.

usage: benchmark_all2all.py [-h] [--dir DIR] [--python PYTHON] arch

positional arguments:
  arch             architecture (dgx1|dgx2)

optional arguments:
  -h, --help       show this help message and exit
  --dir DIR        output directory (default: .)
  --python PYTHON  python interpreter (default: python3)

Note: Takes approximately 22 minutes on a DGX-1 (Volta).

Result

Benchmark files (csv) and plots (pdf) for either DGX-1 or DGX-2 topology.

---

plot_results.py

Plot Benchmark Results

Prequisits

You have already generated one or multiple csv files using benchmark_plan.py or benchmark_all2all.py.

Usage

usage: plot_results.py [-h] [--output OUTPUT] [--reduction REDUCTION]
                       csvs [csvs ...]

positional arguments:
  csvs                  csv traces to be included in the plot e.g.
                        <path%label%color> or <path%label> or <path>

optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
                        output file (default: show plot)
  --reduction REDUCTION, -r REDUCTION
                        reduction operation of multiple runs
                        (median|mean|min|max) (default: median)

Note: % denotes the seperator between filepath, label and color e.g. mypath/myplan.json%mytracelabel%red.

Result

A graph generated by matplotlib showing the collective's achieved throupghput (either displayed or saved).