You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To parallelize the execution of your workflow using the default backend,
19
-
[loky](https://loky.readthedocs.io/), pass an integer greater than 1 or `'auto'` to the
20
-
command-line interface. By default, only one worker is used.
18
+
When the plugin is only installed and pytask executed, the tasks are not run in
19
+
parallel.
20
+
21
+
For parallelization with the default backend [loky](https://loky.readthedocs.io/), you need to launch multiple workers.
22
+
23
+
`````{tab-set}
24
+
````{tab-item} CLI
25
+
:sync: cli
21
26
22
27
```console
23
-
$ pytask -n 2
24
-
$ pytask --n-workers 2
28
+
pytask -n 2
29
+
pytask --n-workers 2
25
30
26
31
# Starts os.cpu_count() - 1 workers.
27
-
$ pytask -n auto
32
+
pytask -n auto
28
33
```
34
+
````
35
+
````{tab-item} Configuration
36
+
:sync: configuration
37
+
38
+
```toml
39
+
[tool.pytask.ini_options]
40
+
n_workers = 2
41
+
42
+
# Starts os.cpu_count() - 1 workers.
43
+
n_workers = "auto"
44
+
```
45
+
````
46
+
`````
29
47
30
48
To use a different backend, pass the `--parallel-backend` option. The following command
31
49
will execute the workflow with one worker and the loky backend.
32
50
51
+
`````{tab-set}
52
+
````{tab-item} CLI
53
+
:sync: cli
54
+
33
55
```console
34
56
pytask --parallel-backend loky
35
57
```
36
-
37
-
The options can also be specified in the configuration file.
58
+
````
59
+
````{tab-item} Configuration
60
+
:sync: configuration
38
61
39
62
```toml
40
63
[tool.pytask.ini_options]
41
-
n_workers = 2
42
64
parallel_backend = "loky"
43
65
```
66
+
````
67
+
`````
44
68
45
69
## Backends
46
70
@@ -54,7 +78,9 @@ If you parallelize the execution of your tasks using two or more workers, do not
54
78
55
79
### loky
56
80
57
-
There are multiple backends available. The default is the backend provided by loky which aims to be a more robust implementation of {class}`multiprocessing.pool.Pool` and in {class}`concurrent.futures.ProcessPoolExecutor`.
81
+
There are multiple backends available. The default is the backend provided by loky which
82
+
aims to be a more robust implementation of {class}`~multiprocessing.pool.Pool` and in
83
+
{class}`~concurrent.futures.ProcessPoolExecutor`.
58
84
59
85
```console
60
86
pytask --parallel-backend loky
@@ -66,14 +92,49 @@ explanation of what CPU- or IO-bound means.)
66
92
67
93
### `concurrent.futures`
68
94
69
-
You can use the values `threads` and `processes` to use the {class}`concurrent.futures.ThreadPoolExecutor` or the {class}`concurrent.futures.ProcessPoolExecutor` respectively.
95
+
You can use the values `threads` and `processes` to use the
96
+
{class}`~concurrent.futures.ThreadPoolExecutor` or the
The `ThreadPoolExecutor` might be an interesting option for you if you have many IO-bound tasks and you do not need to create many expensive processes.
99
+
The {class}`~concurrent.futures.ThreadPoolExecutor` might be an interesting option for
100
+
you if you have many IO-bound tasks and you do not need to create many expensive
101
+
processes.
102
+
103
+
`````{tab-set}
104
+
````{tab-item} CLI
105
+
:sync: cli
72
106
73
107
```console
74
108
pytask --parallel-backend threads
109
+
```
110
+
````
111
+
````{tab-item} Configuration
112
+
:sync: configuration
113
+
114
+
```toml
115
+
[tool.pytask.ini_options]
116
+
parallel_backend = "threads"
117
+
```
118
+
````
119
+
`````
120
+
121
+
`````{tab-set}
122
+
````{tab-item} CLI
123
+
:sync: cli
124
+
125
+
```console
75
126
pytask --parallel-backend processes
76
127
```
128
+
````
129
+
````{tab-item} Configuration
130
+
:sync: configuration
131
+
132
+
```toml
133
+
[tool.pytask.ini_options]
134
+
parallel_backend = "processes"
135
+
```
136
+
````
137
+
`````
77
138
78
139
```{important}
79
140
Capturing warnings is not thread-safe. Therefore, warnings cannot be captured reliably
@@ -82,18 +143,37 @@ when tasks are parallelized with `--parallel-backend threads`.
82
143
83
144
### dask + coiled
84
145
85
-
dask and coiled together provide the option to execute your workflow on cloud providers like AWS, GCP or Azure. Check out the [dedicated guide](dask.md) if you are interested in that.
146
+
dask and coiled together provide the option to execute your workflow on cloud providers
147
+
like AWS, GCP or Azure. Check out the [dedicated guide](dask.md) if you are interested
148
+
in that.
86
149
87
150
Using the default mode, dask will spawn multiple local workers to process the tasks.
88
151
152
+
`````{tab-set}
153
+
````{tab-item} CLI
154
+
:sync: cli
155
+
89
156
```console
90
157
pytask --parallel-backend dask
91
158
```
159
+
````
160
+
````{tab-item} Configuration
161
+
:sync: configuration
162
+
163
+
```toml
164
+
[tool.pytask.ini_options]
165
+
parallel_backend = "dask"
166
+
```
167
+
````
168
+
`````
92
169
93
170
### Custom executors
94
171
95
-
You can also use any custom executor that implements the {class}`concurrent.futures.Executor` interface. Read more about it in [](custom_executors.md).
172
+
You can also use any custom executor that implements the
173
+
{class}`~concurrent.futures.Executor` interface. Read more about it in
174
+
[](custom_executors.md).
96
175
97
176
```{important}
98
-
Please, consider contributing your executor to pytask-parallel if you believe it could be helpful to other people. Start by creating an issue or a draft PR.
177
+
Please, consider contributing your executor to pytask-parallel if you believe it could
178
+
be helpful to other people. Start by creating an issue or a draft PR.
0 commit comments