Parameterization for Workflows - Quiet Nodes?

[TheChymera]

For a while now our lab has been using Python functions which basically construct a nipype workflow and then run it. A major problem is that we are using quite a few variations depending on experiment design, which involve adding or removing a node. Even in such cases where we can make this choice intelligently and automatically based on the input data, and in contrast to simply automating a parameter for a node, adding or not adding a node to a workflow leads to a logic branch. Having multiple optional nodes connected to each other leads to really inelegant nested if statements and/or long lists of multi-conditional elifs.

It would be great if this could be more easily managed. One simple (but hackish) idea I had was to have the option of making nodes quiet - i.e. every node has a quiet attribute, meaning that it simply passes the in_file to out_file, and gets excluded from the DOT graph. This, of course, would only work for nodes having an in_file and out_file.

Is something like this already possible, or is there perhaps an even simpler solution I am not seeing?

@damaggu

[oesteban]

Solving this problem is one of the central goals for the new engine in nipype 2.0, right @satra?

[satra]

@TheChymera - @oesteban is correct that the new engine is supposed to take care of some of these things. however, the details will matter of how and when passthrough happens. in the past, we have done something like this inside a function node. we have realized that there could be generic replacements for some of our function nodes.

at present we (i mean @djarecka ) are focused on mapping/squashing components of the engine that deal with scalability. in particular, we are going to break out the engine into it's own module/package. this will be generic and should be usable across domains.

i will open a google doc where we can discuss all the new features of nipype 2.0. as well as wishlists. i think the broader api should be something that makes it easy to create complex workflows. this may end up being a new language.