Wrap executorlib executors

[liamhuber]

To exploit the new caching interface provided in executorlib-1.5.0 so that nodes can rely on their running state and lexical path to access previously executed results.

Locally with the SingleNodeExecutor everything is looking good, but that doesn't natively support terminating the process that submit the job. I'd like to play around with this using the SlurmClusterExecutor on the cluster before making further changes.

[github-actions]

👈 Launch a binder notebook on branch pyiron/pyiron_workflow/executor

[newfadel]

This is a great step forward, leveraging executorlib-1.5.0's caching interface will be very beneficial for performance and result reproducibility!

Regarding your work with SlurmClusterExecutor, it's definitely the right approach to test process termination in a real cluster environment. Once you've had a chance to experiment, I'd be particularly interested in understanding:

1. Process Management and Cleanup: How will SlurmClusterExecutor manage the lifecycle of the submitted jobs on the cluster? Specifically, what mechanisms will be in place to ensure proper termination and cleanup of resources, especially if the submitting Python process exits unexpectedly or is terminated?
2. Caching Strategy with Distributed Execution: With the caching now active across potentially multiple nodes, have you considered potential challenges around cache invalidation or consistency if underlying data changes or if a previous computation failed? Are there plans to implement strategies to ensure the cached results remain valid and reliable in a distributed setting?
3. Error Handling and Robustness: For production use on a cluster, robust error handling is crucial. How will the wrapped executors handle common cluster-specific issues like job failures, network interruptions, or resource limits?

Looking forward to seeing the progress on this! 😄

[codecov]

Codecov Report

Attention: Patch coverage is 88.88889% with 9 lines in your changes missing coverage. Please review.

Project coverage is 92.05%. Comparing base (4841e34) to head (f579159).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pyiron_workflow/node.py	70.00%	6 Missing ⚠️
pyiron_workflow/executors/wrapped_executorlib.py	96.29%	1 Missing ⚠️
pyiron_workflow/mixin/run.py	96.42%	1 Missing ⚠️
pyiron_workflow/nodes/composite.py	66.66%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (88.88%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #678      +/-   ##
==========================================
- Coverage   92.11%   92.05%   -0.07%     
==========================================
  Files          33       34       +1     
  Lines        3665     3725      +60     
==========================================
+ Hits         3376     3429      +53     
- Misses        289      296       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bot spam

[liamhuber]

This is currently working to all attempted behaviour when run together with pyiron/executorlib#712

[liamhuber]

ImportError: cannot import name 'TestClusterExecutor' from 'executorlib.api' (/home/runner/work/pyiron_workflow/pyiron_workflow/cached-miniforge/my-env/lib/python3.13/site-packages/executorlib/api.py). Did you mean: 'FluxClusterExecutor'?

Waiting on executorlib-1.5.3

[liamhuber]

Together with pyiron/executorlib#732 this is working very nicely on the cluster now. I can

• Let the workflow run
• Interrupt the workflow after the slurm job has started
  • Restart the workflow after the slurm job is complete
  • Restart the workflow while the slurm job is still going

and in all cases everything runs perfectly smoothly. I.e. I can start with this:

import pyiron_workflow as pwf
from pyiron_workflow.executors.wrapped_executorlib import CacheSlurmClusterExecutor


wf = pwf.Workflow("executor_test")
wf.n1 = pwf.std.UserInput(20)
wf.n2 = pwf.std.Sleep(wf.n1)
wf.n3 = pwf.std.UserInput(wf.n2)

wf.n2.executor = (CacheSlurmClusterExecutor, (), {"resource_dict": {"partition": "s.cmfe"}})

wf.run()

And then either let it run, or restart the kernel and follow up with this after the appropriate delay for the case I'm interested in:

import pyiron_workflow as pwf
from pyiron_workflow.executors.wrapped_executorlib import CacheSlurmClusterExecutor

wf = pwf.Workflow("executor_test")
wf.load(filename=wf.label + "/recovery.pckl")

wf.failed = False
wf.use_cache = False
wf.run()

Outside the scope of this PR but on the TODO list is:

• Convenience around use_cache and Workflow use_cache is annoying on composites #699
• Convenience around running workflows in a background thread so I can make a real save and not rely on a recovery file Let workflows be easily run in the background #700
• CI tests on Slurm ala Test with SLURM executorlib#726 (done and merged in Draft a slurm test #704)
• Adding the new wrapped executorlib.SlurmClusterExecutor to the API Expose executorlib SLURM power in API #708
• Removing the old way of finding serialized slurm results Remove old caching tools #709

[liamhuber]

Codecov complains that Node._clean_wrapped_executorlib_executor_cache is not being tested, but it's just flat out wrong. It is absolutely being invoked in tests.integration.test_wrapped_executorlib.TestWrappedExecutorlib.test_automatic_cleaning.

[liamhuber]

• CI tests on Slurm

Actually, I'd be more comfortable with this PR if it included these. I'll still leave exposure in the API for later, but let's take a crack at robust testing right here.