Skip to content

Commit 6798d01

Browse files
AlexsandrussAlexsandruss
committed
Update sklbench args info
1 parent 765a5c3 commit 6798d01

File tree

4 files changed

+32
-30
lines changed

4 files changed

+32
-30
lines changed

‎README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -53,13 +53,13 @@ python -m sklbench --config configs/sklearn_example.json
5353
The default output is a file with JSON-formatted results of benchmarking cases. To generate a better human-readable report, use the following command:
5454

5555
```bash
56-
python -m sklbench --config configs/sklearn_example.json --generate-report
56+
python -m sklbench --config configs/sklearn_example.json --report
5757
```
5858

5959
By default, output and report file paths are `result.json` and `report.xlsx`. To specify custom file paths, run:
6060

6161
```bash
62-
python -m sklbench --config configs/sklearn_example.json --generate-report --result-file result_example.json --report-file report_example.xlsx
62+
python -m sklbench --config configs/sklearn_example.json --report --result-file result_example.json --report-file report_example.xlsx
6363
```
6464

6565
For a description of all benchmarks runner arguments, refer to [documentation](sklbench/runner/README.md#arguments).

‎sklbench/report/README.md

+13-12
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,19 @@ Raw results are converted into a pandas dataframe and the final report is made b
1111
## Arguments
1212
<!-- Note: generate arguments table using runner: `python -m sklbench --describe-parser` -->
1313

14-
| Name | Type | Default value | Choices | Description |
15-
|:-----------------------------------|:------|:------------------------------------------|:-----------------------------------------------|:--------------------------------------------------------------------------------------|
16-
| `--report-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for report generator. |
17-
| `--result-files` | str | ['result.json'] | | Result file path[s] from scikit-learn_bench runs for report generation. |
18-
| `--report-file` | str | report.xlsx | | Report file path. |
19-
| `--report-type` | str | separate-tables | ('separate-tables',) | Report type ("separate-tables" is the only supported now). |
20-
| `--drop-columns`</br>`--drop-cols` | str | [] | | Columns to drop from report. |
21-
| `--diff-columns`</br>`--diff-cols` | str | ['environment_name', 'library', 'device'] | | Columns to show difference between. |
22-
| `--split-columns` | str | ['estimator', 'method', 'function'] | | Splitting columns for subreports/sheets. |
23-
| `--diffs-selection` | str | upper_triangle | ['upper_triangle', 'lower_triangle', 'matrix'] | Selects which part of one-vs-one difference to show (all matrix or one of triangles). |
24-
| `--time-color-scale` | float | [0.8, 1.0, 10.0] | | Time improvement color scale in report. |
25-
| `--metric-color-scale` | float | [0.99, 0.995, 1.01] | | Metric improvement color scale in report. |
14+
| Name | Type | Default value | Choices | Description |
15+
|:-----------------------------------------------|:-------|:----------------------------------------------------|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
16+
| `--report-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for report generator. |
17+
| `--result-files` | str | ['result.json'] | | Result file path[s] from scikit-learn_bench runs for report generation. |
18+
| `--report-file` | str | report.xlsx | | Report file path. |
19+
| `--report-type` | str | separate-tables | ('separate-tables',) | Report type ("separate-tables" is the only supported now). |
20+
| `--compatibility-mode` | | False | | [EXPERIMENTAL] Compatibility mode drops and modifies results to make them comparable (for example, sklearn and cuML parameters). |
21+
| `--drop-columns`</br>`--drop-cols` | str | [] | | Columns to drop from report. |
22+
| `--diff-columns`</br>`--diff-cols` | str | ['environment_name', 'library', 'format', 'device'] | | Columns to show difference between. |
23+
| `--split-columns` | str | ['estimator', 'method', 'function'] | | Splitting columns for subreports/sheets. |
24+
| `--diffs-selection` | str | upper_triangle | ['upper_triangle', 'lower_triangle', 'matrix'] | Selects which part of one-vs-one difference to show (all matrix or one of triangles). |
25+
| `--perf-color-scale` | float | [0.8, 1.0, 10.0] | | Color scale for performance metric improvement in report. |
26+
| `--quality-color-scale` | float | [0.99, 0.995, 1.01] | | Color scale for quality metric improvement in report. |
2627

2728
---
2829
[Documentation tree](../../README.md#-documentation-tree)

‎sklbench/runner/README.md

+16-15
Original file line numberDiff line numberDiff line change
@@ -37,21 +37,22 @@ flowchart LR
3737
## Arguments
3838
<!-- Note: generate arguments table using runner: `python -m sklbench --describe-parser` -->
3939

40-
| Name | Type | Default value | Choices | Description |
41-
|:-----------------------------------------------|:-----|:--------------|:--------------------------------------|:-------------------------------------------------------------------------------------------------------|
42-
| `--runner-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for benchmarks runner. |
43-
| `--bench-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for each running benchmark. |
44-
| `--log-level` | str | | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Global logging level for benchmarks: overwrites runner, bench and report log levels. |
45-
| `--config`</br>`--configs` | str | | | Paths to a configuration files or/and directories that contain configuration files. |
46-
| `--parameters`</br>`--params`</br>`-p` | str | | | Globally defines or overwrites config parameters. For example: `-p data:dtype=float32 data:order=F`. |
47-
| `--parameter-filters`</br>`--filters`</br>`-f` | str | | | Filters benchmark bench_cases by config parameters. For example: `-f data:dtype=float32 data:order=F`. |
48-
| `--result-file`</br>`-r` | str | result.json | | File path to store scikit-learn_bench's runned cases results. |
49-
| `--environment-alias`</br>`--env-alias` | str | | | Environment alias to use instead of it's configuration hash. |
50-
| `--prefetch-datasets` | Flag | False | | [EXPERIMENTAL] Load datasets in parallel before running benchmarks. |
51-
| `--describe-parser` | Flag | False | | Print parser description in Markdown table format and exit. |
52-
| `--generate-report`</br>`--gen-report` | Flag | False | | Enables generation of report. |
53-
54-
Also, benchmarks runner includes and takes into the account [arguments of report generator](../report/README.md#arguments) if `--generate-report` flag is passed.
40+
| Name | Type | Default value | Choices | Description |
41+
|:-----------------------------------------------|:-------|:----------------------------------------------------|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
42+
| `--runner-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for benchmarks runner. |
43+
| `--bench-log-level` | str | WARNING | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Logging level for each running benchmark. |
44+
| `--log-level`</br>`-l` | str | | ('ERROR', 'WARNING', 'INFO', 'DEBUG') | Global logging level for benchmarks: overwrites runner, benchmarks and report logging levels. |
45+
| `--config`</br>`--configs`</br>`-c` | str | | | Paths to a configuration files or/and directories that contain configuration files. |
46+
| `--parameters`</br>`--params`</br>`-p` | str | | | Globally defines or overwrites config parameters. For example: `-p data:dtype=float32 data:order=F`. |
47+
| `--parameter-filters`</br>`--filters`</br>`-f` | str | | | Filters benchmarking cases by parameter values. For example: `-f data:dtype=float32 data:order=F`. |
48+
| `--result-file`</br>`-r` | str | result.json | | File path to store scikit-learn_bench's runned cases results. |
49+
| `--environment-name`</br>`--env-name`</br>`-e` | str | | | Environment name to use instead of it's configuration hash. |
50+
| `--prefetch-datasets` | | False | | Load all requested datasets in parallel before running benchmarks. |
51+
| `--exit-on-error` | | False | | Interrupt runner and exit if last benchmark failed with error. |
52+
| `--describe-parser` | | False | | Print parser description in Markdown table format and exit. |
53+
| `--report` | | False | | Enables generation of report. |
54+
55+
Also, benchmarks runner includes and takes into the account [arguments of report generator](../report/README.md#arguments) if `--report` flag is passed.
5556

5657
---
5758
[Documentation tree](../../README.md#-documentation-tree)

‎sklbench/runner/arguments.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ def add_runner_arguments(parser: argparse.ArgumentParser) -> argparse.ArgumentPa
7575
type=str,
7676
choices=("ERROR", "WARNING", "INFO", "DEBUG"),
7777
help="Global logging level for benchmarks: "
78-
"overwrites runner, bench and report logging levels.",
78+
"overwrites runner, benchmarks and report logging levels.",
7979
)
8080
# benchmarking cases finding, overwriting and filtering
8181
parser.add_argument(

0 commit comments

Comments
 (0)