Use reference to err to avoid deletion in Py3.4 #1551

ashgillman · 2016-07-27T02:43:31Z

Update on #1446, I was still getting an error on Python 3.4.4

See also PR #1447

coveralls · 2016-07-27T02:52:18Z

Coverage increased (+0.0006%) to 72.326% when pulling 8b3c8df on ashgillman:FixMapNodePy3.4 into 3ed411b on nipy:master.

chrisgorgo · 2016-08-30T19:00:55Z

@oesteban how does this relate to your py3 efforts?

oesteban · 2016-08-30T19:04:55Z

In Circle we were not testing python 3.4. In travis it didn't complain. I guess there is no test to exercise that exception catching.

ashgillman · 2016-09-08T23:25:31Z

Here is an MWE to demonstrate my issue.

from nipype.pipeline.engine import MapNode, Workflow
from nipype.interfaces import Function

def subworkflow_wrapper(arg):
    import os
    from nipype.pipeline.engine import Node, Workflow
    from nipype.interfaces import Function
    from nipype.utils.filemanip import loadpkl

    crash_program = """\
def crash_program(arg):
    raise Exception('Important message, {}'.format(arg))
"""
    sub_wf = Workflow(name='sub_wf')
    sub_wf.base_dir = '.'
    crash = Node(Function(function_str=crash_program,
                          input_names=['arg'],
                          output_names=['res']),
                name='crash')
    sub_wf.add_nodes([crash])

    sub_wf.inputs.crash.arg = arg

    sub_wf.run()

    results = loadpkl(
        os.path.join(sub_wf.name, crash.name,
                    'result_{}.pklz'.format(crash.name)))
    return results.inputs['_outputs']['res']

wf = Workflow(name='wf')
sub_wf_each = MapNode(
    Function(function=subworkflow_wrapper,
            input_names=['arg'],
            output_names=['res']),
    iterfield=['arg'],
    name='sub_wf_each')

wf.add_nodes((sub_wf_each, ))
wf.inputs.sub_wf_each.arg = [1, 2, 3]
wf.run()

Th important part of the log being:

160909-09:23:31,375 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/gdq8zym3pd538si7mrfp4av14zggis1i-python3.4-nipype-0.12.0/lib/python3.4/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
    node.run(updatehash=updatehash)
  File "/nix/store/gdq8zym3pd538si7mrfp4av14zggis1i-python3.4-nipype-0.12.0/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 394, in run
    self._run_interface()
  File "/nix/store/gdq8zym3pd538si7mrfp4av14zggis1i-python3.4-nipype-0.12.0/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1306, in _run_interface
    updatehash=updatehash))
  File "/nix/store/gdq8zym3pd538si7mrfp4av14zggis1i-python3.4-nipype-0.12.0/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1169, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/gdq8zym3pd538si7mrfp4av14zggis1i-python3.4-nipype-0.12.0/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1162, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

This patch (replicating the PR) fixes it:

import nipype
from nipype.utils.misc import str2bool
def _node_runner(self, nodes, updatehash=False):
    for i, node in nodes:
        err = None
        try:
            node.run(updatehash=updatehash)
        except Exception as this_err:
            err = this_err # save reference
            if str2bool(self.config['execution']['stop_on_first_crash']):
                self._result = node.result
                raise
        finally:
            yield i, node, err
nipype.pipeline.engine.nodes.MapNode._node_runner = _node_runner

ashgillman · 2016-09-08T23:31:12Z

I realised I was using 12.0 in that last example, but the same error holds for 12.1

160909-09:27:35,148 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
    node.run(updatehash=updatehash)
  File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 394, in run
    self._run_interface()
  File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1306, in _run_interface
    updatehash=updatehash))
  File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1169, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1162, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

Similarly for Py3.3

160909-09:29:43,881 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
    node.run(updatehash=updatehash)
  File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 394, in run
    self._run_interface()
  File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1306, in _run_interface
    updatehash=updatehash))
  File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1169, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1162, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

chrisgorgo · 2016-09-08T23:35:04Z

Could you try current master? A lot of py3 improvements have been fixed
recently.

On Thu, Sep 8, 2016 at 4:31 PM, Ashley Gillman [email protected]
wrote:

I realised I was using 12.0 in that last example, but the same error holds
for 12.1

160909-09:27:35,148 workflow INFO:
Traceback (most recent call last):
File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
node.run(updatehash=updatehash)
File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 394, in run
self._run_interface()
File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1306, in _run_interface
updatehash=updatehash))
File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1169, in _collate_results
for i, node, err in nodes:
File "/nix/store/qa2c825midsqi30k0kkz68klw67yp8d7-python3.4-nipype-0.12.1/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1162, in _node_runner
yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

Similarly for Py3.3

160909-09:29:43,881 workflow INFO:
Traceback (most recent call last):
File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
node.run(updatehash=updatehash)
File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 394, in run
self._run_interface()
File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1306, in _run_interface
updatehash=updatehash))
File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1169, in _collate_results
for i, node, err in nodes:
File "/nix/store/pfdgknf2q99j3rb3qgr4f69a9ynn2zl7-python3.3-nipype-0.12.1/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1162, in _node_runner
yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1551 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAOkp4PE4Fv-pR_N_vOV7L06FdGYtaBSks5qoJrAgaJpZM4JVyIY
.

ashgillman · 2016-09-09T01:31:04Z

On master (0f9a620)

160909-10:20:32,360 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/mgzabyx9n2fhdg9zfb276f0dgzjmd7s4-python3.3-nipype-master/lib/python3.3/site-packages/nipype/pipeline/plugins/linear.py", line 43, in run
    node.run(updatehash=updatehash)
  File "/nix/store/mgzabyx9n2fhdg9zfb276f0dgzjmd7s4-python3.3-nipype-master/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 377, in run
    self._run_interface()
  File "/nix/store/mgzabyx9n2fhdg9zfb276f0dgzjmd7s4-python3.3-nipype-master/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1294, in _run_interface
    updatehash=updatehash))
  File "/nix/store/mgzabyx9n2fhdg9zfb276f0dgzjmd7s4-python3.3-nipype-master/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1157, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/mgzabyx9n2fhdg9zfb276f0dgzjmd7s4-python3.3-nipype-master/lib/python3.3/site-packages/nipype/pipeline/engine/nodes.py", line 1150, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

160909-10:34:17,162 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/95dlwka1jjbzca2fz232m6rwi1453svr-python3.4-nipype-master/lib/python3.4/site-packages/nipype/pipeline/plugins/linear.py", line 43, in run
    node.run(updatehash=updatehash)
  File "/nix/store/95dlwka1jjbzca2fz232m6rwi1453svr-python3.4-nipype-master/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 377, in run
    self._run_interface()
  File "/nix/store/95dlwka1jjbzca2fz232m6rwi1453svr-python3.4-nipype-master/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1294, in _run_interface
    updatehash=updatehash))
  File "/nix/store/95dlwka1jjbzca2fz232m6rwi1453svr-python3.4-nipype-master/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1157, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/95dlwka1jjbzca2fz232m6rwi1453svr-python3.4-nipype-master/lib/python3.4/site-packages/nipype/pipeline/engine/nodes.py", line 1150, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

160909-11:22:58,974 workflow INFO:
     Traceback (most recent call last):
  File "/nix/store/a58j94iacgx2ip52y1i6ff6n7dq3fpdb-python3.5-nipype-master/lib/python3.5/site-packages/nipype/pipeline/plugins/linear.py", line 43, in run
    node.run(updatehash=updatehash)
  File "/nix/store/a58j94iacgx2ip52y1i6ff6n7dq3fpdb-python3.5-nipype-master/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py", line 377, in run
    self._run_interface()
  File "/nix/store/a58j94iacgx2ip52y1i6ff6n7dq3fpdb-python3.5-nipype-master/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py", line 1294, in _run_interface
    updatehash=updatehash))
  File "/nix/store/a58j94iacgx2ip52y1i6ff6n7dq3fpdb-python3.5-nipype-master/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py", line 1157, in _collate_results
    for i, node, err in nodes:
  File "/nix/store/a58j94iacgx2ip52y1i6ff6n7dq3fpdb-python3.5-nipype-master/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py", line 1150, in _node_runner
    yield i, node, err
UnboundLocalError: local variable 'err' referenced before assignment

I hadn't realised it was all the Py3 versions. It seems strange no one else has noticed this, can any one else replicate?

alexsavio · 2016-10-21T14:08:36Z

Yes, the same happens to me. I didn't see it, since there is not test for this. Could you make a test from your MWE and push it along your fix?

ashgillman · 2016-10-24T07:47:00Z

Test added, fails w/o change, passes with new change.

Use reference to err to avoid deletion in Py3.4

8b3c8df

ashgillman mentioned this pull request Jul 27, 2016

Strange return values in nodes.py #1446

Closed

ashgillman added 2 commits October 24, 2016 17:40

Add test case demonstrating py3 failure on wrapped workflow mapnode

d794f02

Merge branch 'master' of github.com:nipy/nipype into FixMapNodePy3.4

a28021c

satra added a commit to satra/nipype that referenced this pull request Dec 18, 2016

fix: addresses issue nipy#1446 (solution from @ashgillman in nipy#1551)

5b0e74c

ashgillman closed this Jan 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use reference to err to avoid deletion in Py3.4 #1551

Use reference to err to avoid deletion in Py3.4 #1551

Uh oh!

ashgillman commented Jul 27, 2016

Uh oh!

coveralls commented Jul 27, 2016 •

edited

Loading

Uh oh!

chrisgorgo commented Aug 30, 2016

Uh oh!

oesteban commented Aug 30, 2016

Uh oh!

ashgillman commented Sep 8, 2016

Uh oh!

ashgillman commented Sep 8, 2016

Uh oh!

chrisgorgo commented Sep 8, 2016

Uh oh!

ashgillman commented Sep 9, 2016

Uh oh!

alexsavio commented Oct 21, 2016

Uh oh!

ashgillman commented Oct 24, 2016

Uh oh!

Uh oh!

Use reference to err to avoid deletion in Py3.4 #1551

Use reference to err to avoid deletion in Py3.4 #1551

Uh oh!

Conversation

ashgillman commented Jul 27, 2016

Uh oh!

coveralls commented Jul 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrisgorgo commented Aug 30, 2016

Uh oh!

oesteban commented Aug 30, 2016

Uh oh!

ashgillman commented Sep 8, 2016

Uh oh!

ashgillman commented Sep 8, 2016

Uh oh!

chrisgorgo commented Sep 8, 2016

Uh oh!

ashgillman commented Sep 9, 2016

Uh oh!

alexsavio commented Oct 21, 2016

Uh oh!

ashgillman commented Oct 24, 2016

Uh oh!

Uh oh!

coveralls commented Jul 27, 2016 •

edited

Loading