Data models

[liamhuber]

TODO: tests done

Adds pydantic models for the atomic¹ and workflow node types.
Edges are structured as dictionaries per #65, and workflows explicitly specify their IO labels per #63.

Out of scope (will stack as a separate PRs):

• Parsing python into these types
• Parsers-as-decorators
• Other graph elements (for, ...)
• Metadata and things pertaining to node store, etc.

Footnotes

1. I'm running with "atomic" right now since both Sam and I are fine with it, but this is subject to change. ↩

[github-actions]

👈 Launch a binder notebook on branch pyiron/flowrep/data_models

[codecov]

Codecov Report

❌ Patch coverage is 97.22222% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.98%. Comparing base (53bb709) to head (298e3e6).

Files with missing lines	Patch %	Lines
flowrep/model.py	97.22%	3 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           type_names      #69      +/-   ##
==============================================
+ Coverage       95.50%   95.98%   +0.48%     
==============================================
  Files               4        4              
  Lines             667      773     +106     
==============================================
+ Hits              637      742     +105     
- Misses             30       31       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liamhuber]

@XzzX, I lifted almost verbatim a couple pydantic snippets from your python workflow definition model file, so I added you as a co-author on those commits. LMK if you'd prefer to handle this a different way.

[liamhuber]

Ok, I'm happy with this. I extended the data model to always include the inputs and outputs per this comment. The next thing I'd like to stack onto this work is to actually import the fully qualified names and validate the node model against the ast inspection of the functions. This would be either a pre-cursor to or part-of writing a parser to go from a python function object to the recipe model (and from there the decorator is a trivial step).

I really like the file serialization helpers @XzzX made here and think they'd be great on the base NodeModel class, but will leave that to another PR.

[liamhuber]

And here's a little example I've been playing with to look at the "json" and "python" dump formats

from flowrep import model

def plus_minus(x, y):
    return x + y, x - y

def sum(x, y):
    return x + y

g = model.WorkflowNode(
    inputs=["x", "y"],
    outputs=["z"],
    nodes={
        "macro": model.WorkflowNode(
            inputs=["a", "b"],
            outputs=["c", "d"],
            nodes={
                "differential": model.AtomicNode(
                    fully_qualified_name="__main__.plus_minus",
                    inputs=["x", "y"],
                    outputs=["output_0", "output_1"],
                )
            },
            edges={
                ("differential", "x"): "a",
                ("differential", "y"): "b",
                "c": ("differential", "output_0"),
                "d": ("differential", "output_1"),
            }
        ),
        "add": model.AtomicNode(
            fully_qualified_name="__main__.add",
            inputs=["x", "y"],
            outputs=["sum"],
        )
    },
    edges={
        ("macro", "a"): "x",
        ("macro", "b"): "y",
        ("add", "x"): ("macro", "c"),
        ("add", "y"): ("macro", "d"),
        "z": ("add", "sum"),
    }
)

jdict = g.model_dump(mode="json")
jdict

[XzzX]

I know this is verbose, but I'd prefer something like this.

class EdgeModel(BaseModel):
    source_node: str  # reference to a node from the nodes list
    source_port: int | None  # either port index or None for the node itself
    target_node: str  # reference to a node from the nodes list
    target_port: str  # name of the target port

[liamhuber]

I can live with this, but do you intentionally exclude the information for graph parents to negotiate IO with their children? Maybe we ought to store that somewhere other than "edges", or the type hints here get very liberal and there are source/target combinations which are disallowed.

[liamhuber]

Actually, I can live with this only under duress -- I still prefer if the data format automatically encodes the restriction that each target can have only one port. What about

class SourceModel(BaseModel):
    source_node: str  # reference to a node from the nodes list
    source_port: int | None  # either port index or None for the node itself

class TargetModel(BaseModel):
    target_node: str  # reference to a node from the nodes list
    target_port: str  # name of the target port

with edges: dict[TargetModel, SourceModel].

Breaking out IO handling, we'd get something like

class InputModel(BaseModel):
    port: str

class OutputModel(BaseModel):
    port: int

and input_transfer: dict[TargetModel, InputModel]; output_transfer: dict[OutputModel, SourceModel] (or whatever we call the variable). Or something similar to this.