Commit 13f2259
authored
refactor(core): unify operation execution pathways and remove unused logic (#220)
* refactor: split orchestrate to enhance maintainability
* fix: typing
* refactor: provide functions via orchestrate
The refactoring moves general/explore orchestration function to private modules for clarity.
However we don't want to expose this private modules external - want to keep the access point orchestrate.py
* feat(core): Convert RayTaskError to original error
* docs(core): Update exceptions raised
orchestrate_explore_operation
* feat(core): Improve granularity of exception handling
* chore(core): merge log change from main
* refactor: update to new import
* refactor: expose required members via orchestrate
* fix: Access via public interface
* refactor: orchestrate params
Remove unused parameters.
* refactor: remove project_context arg
Not needed as its we must use the one given by discover_space.project_context
* refactor(orchestration): remove queue arg
from orchestrate and orchestrate_exploer_operation.
it was never not None so not required to be a parameter
* refactor(orchestration): rename param
execid -> namespace
* refactor(orchestration): remove queue arg
* refactor(orchestration): Remove BaseOperationRunConfiguration
Originally required as it held common function for two sub-classes.
However, one of those was removed meaning we don't need a base class anymore and can simplify the code
* chore(comments): delete comment
* refactor(orchestrate): move acquiring ac_config
From orchestrate to general_orchestration.
This is because
- orchestrate_explore_operation already does this so it doesn't need it from orchestrate
- the information is not passed from orchestrate to orchestrate_general_operation
- orchestrate_general_operation should have this info but it needs to acquire it itself in case the function is called directly (not via orchestrate)
Hence, it's redundant to have this code here. The correct place its needed is in orchestrate_general_operation
* refactor(orchestrate): actuator configuration validation
Extract functions for getting and validating actuator configurations. Rename methods.
The reason is that as-is simplified get/validate is tied to DiscoveryOperationResourceConfiguration but this constrains where it has to happen to the existence of an instance of the class.
* fix: imports
* fix: temporarily use private module
* refactor(orchestration): add_operation_from_configuration_to_metastore
now takes more basic components and is called create_operation_and_add_to_metastore
* refactor(orchestration): _run_operation_harness
Update parameters to take operator module/parameters/info rather than having them bundled in DiscoveryOperationConfiguration instance
Update to new create_operation_and_add_to_metastore interface.
* refactor(orchestration): orchestrate_explore_operation
Update parameters to take operator module/parameters/info rather than a DiscoveryOperationResourceConfiguration instance.
This is so the parameters are as similar as possible to orchestrate_general_operation make the paths to each more unififed.
Remove namespace parameter = the orchestrate function will generate its own namespace for its operation
Simplify return to OperationOutput - other instance returned are deducible from this or were input
Remove orchestrate_operation_function_wrapper - no longer necessary due to change in orchestrate_explore_operation input/output
* refactor(orchestration): setup operator
Change parameters to take operator_module and parameters instead of requiring them to be bundled in DiscoveryOperationResourceConfiguration
* fix(orchestration): Check operation_info namespace
* feat(core): Add namespace field to FunctionOperation
To allow communicating it to a general operation that wants to create ray actors/remotes
* refactor(orchestrate): update to new interface
No need to call wrapper or set namespace explicitly.
* fix(orchestrate): Pass namespace to general operators
Also update to new run_operation_harness
* refactor(orchestrate): Update orchestrate
- Don't namespace at orchestrate level, let the orchestration functions set namespace
- Update to new orchestrate_*_operation interfaces
- Remove orchestrate_operation_function - no longer necessary
* refactor(orchestrate): Remove unused parameter
* docs(operator): update to new function
* docs(orchestrate): update docstrings
* chore(core): apply suggestions from code review
* chore(core): change field name
---------
Signed-off-by: Michael Johnston <[email protected]>1 parent ece2f7d commit 13f2259
File tree
14 files changed
+452
-391
lines changed- orchestrator
- cli/resources/operation
- core/operation
- modules/operators
- utilities
- plugins/operators/ray_tune/ado_ray_tune
- tests/core
- website/docs/operators
14 files changed
+452
-391
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
92 | 91 | | |
93 | 92 | | |
94 | 93 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
41 | 175 | | |
42 | 176 | | |
43 | 177 | | |
| |||
128 | 262 | | |
129 | 263 | | |
130 | 264 | | |
131 | | - | |
132 | | - | |
133 | | - | |
134 | | - | |
| 265 | + | |
| 266 | + | |
135 | 267 | | |
136 | 268 | | |
137 | 269 | | |
| |||
140 | 272 | | |
141 | 273 | | |
142 | 274 | | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
143 | 279 | | |
144 | 280 | | |
145 | 281 | | |
| |||
160 | 296 | | |
161 | 297 | | |
162 | 298 | | |
163 | | - | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
164 | 302 | | |
165 | 303 | | |
166 | | - | |
167 | | - | |
168 | 304 | | |
169 | 305 | | |
170 | 306 | | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
198 | 310 | | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | 311 | | |
243 | 312 | | |
244 | 313 | | |
245 | 314 | | |
| 315 | + | |
246 | 316 | | |
247 | | - | |
| 317 | + | |
248 | 318 | | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
| 319 | + | |
| 320 | + | |
255 | 321 | | |
256 | | - | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
262 | 325 | | |
263 | | - | |
264 | 326 | | |
| 327 | + | |
| 328 | + | |
265 | 329 | | |
266 | | - | |
267 | | - | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
268 | 335 | | |
269 | | - | |
270 | | - | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
271 | 339 | | |
272 | 340 | | |
273 | 341 | | |
274 | 342 | | |
275 | 343 | | |
276 | 344 | | |
277 | 345 | | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
0 commit comments