python 2.7 string issue in plugins #1621

bpinsard · 2016-09-12T19:35:39Z

with recent updates, I get the following error using SGE plugin:

TraitError: Each key of the 'environ' trait of a CommandLineInputSpec instance must be a newstr or None, but a value of 'MDMSESSION' <type 'str'> was specified.

It seems that the traits definition uses the future newstr which does not match with what os.environ contains (old str). It is certainly the same for other plugins, and this is the only places where environ is fed to CommandLine in whole nipype.

Any idea how to fix this?
Thanks

The text was updated successfully, but these errors were encountered:

oesteban · 2016-09-12T21:50:10Z

Ok, I'll look at this now

Fixes nipy#1621

bpinsard · 2016-09-13T20:37:11Z

Thanks but in fact in seems something more global, the future newstr seems to interact with all traits string type checking. Each string that comes from os.environ causes crash.

n_tesselate = pe.Node(
    freesurfer.MRITessellate(label_value=1),
    name='tesselate')

TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of u'SUBJECTS_DIR' <type 'unicode'> was specified

I do not know if there is a simple way to change this except removing all future

oesteban · 2016-09-13T20:40:30Z

That's the thing: without unicode_literals from __future__ and the newstr we break python 3 compatibility. We've dealt with this issue in most of the situations, but of course some are not tested (our coverage is around 70% now, we want to raise that).

bpinsard · 2016-09-13T20:43:55Z

ok, I am just surprised that it got merged in master considering that it breaks a lot of nipype interfaces/plugins when used in 2.x environment. Or am I the only one to experience problems with 2.x?

oesteban · 2016-09-13T20:45:52Z

It doesn't break a lot nipype interfaces. We are constantly checking the circleCI examples for both 2.7 and 3.5 (https://circleci.com/gh/nipy/nipype/1475).

The alternative would be stick with 2.7 forever.

bpinsard · 2016-09-13T21:08:42Z

I have been running nosetest on nipype in my freshly cleaned conda 2.7 environment, and in fact it seems that any inputs that requires string from os.environ (most inheriting from CommandLine) crashes, which is a huge problem for most pipelines that I use.
I am preparing a PR to change DictStrStr

-DictStrStr = traits.Dict(str, (bytes, str))
+DictStrStr = traits.Dict((bytes,str,unicode), (bytes, str,unicode))

This will solve my case for environ problem, nosetests crashes for unicode string in other places, but I am less concerned about that for now.

chrisgorgo · 2016-09-13T21:11:56Z

This is weird... Are you testing current master? Travis was suppose to run tests in python 2.7, 3.4 and 3.5 to ensure all of those are supported. See: https://travis-ci.org/nipy/nipype

bpinsard · 2016-09-13T21:16:39Z

yes I am testing the current master

bpinsard · 2016-09-13T21:17:16Z

Since I updated, I have tested on my regular python and also anaconda.

chrisgorgo · 2016-09-13T21:22:28Z

I wonder if this could have something to do with the version of futures/six on your system.

bpinsard · 2016-09-13T21:25:42Z

six==1.10.0
future==0.15.2

chrisgorgo · 2016-09-13T21:29:18Z

It's the same as on Travis :/ https://s3.amazonaws.com/archive.travis-ci.org/jobs/159706130/log.txt

oesteban · 2016-09-13T21:32:08Z

With nipype installed from conda and updated with current master:

(nipypedev-2.7) oesteban@dendrite:~/workspace/mriqc$ python --version
Python 2.7.11 :: Continuum Analytics, Inc.
(nipypedev-2.7) oesteban@dendrite:~/workspace/mriqc$ LANG=en_US.C; LC_ALL=en_US.C; python -c "from nipype.pipeline import engine as pe; from nipype.interfaces import freesurfer; n_tesselate = pe.Node(freesurfer.MRITessellate(label_value=1), name='tesselate')"

oesteban · 2016-09-13T21:36:49Z

@bpinsard may you hand over me a simple workflow that fails for you, using the SGE plugin? I have already set up a Docker image with SGE and I can try to replicate the problem. I will think of some tests for the SGE plugin as well. Current coverage is not very comprehensive:

nipype/pipeline/plugins/sge.py                                                   232    191     66      0    15%
nipype/pipeline/plugins/sgegraph.py                                               89     77     36      0    10%

bpinsard · 2016-09-14T14:14:40Z

just to show the output of the command line above

python -c "from nipype.pipeline import engine as pe; from nipype.interfaces import freesurfer; n_tesselate = pe.Node(freesurfer.MRITessellate(label_value=1), name='tesselate')"
Exception occurred in traits notification handler.
Please check the log file for details.
Exception occurred in traits notification handler for object: 
args = <undefined>
environ = {}
ignore_exception = False
in_file = <undefined>
label_value = 1
out_file = <undefined>
subjects_dir = /home/bpinsard/softs/freesurfer/subjects
terminal_output = stream
tesselate_all_voxels = <undefined>
use_real_RAS_coordinates = <undefined>
, trait: subjects_dir, old value: <undefined>, new value: /home/bpinsard/softs/freesurfer/subjects
Traceback (most recent call last):
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 520, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 483, in dispatch
    handler( *args )
  File "nipype/interfaces/freesurfer/base.py", line 116, in _subjects_dir_update
    self.inputs.subjects_dir})
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3206, in update
    new_dic = self._validate_dic( dic )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3304, in _validate_dic
    raise excp
TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of u'SUBJECTS_DIR' <type 'unicode'> was specified.
160914-09:55:18,496 traits ERROR:
     Exception occurred in traits notification handler for object: 
args = <undefined>
environ = {}
ignore_exception = False
in_file = <undefined>
label_value = 1
out_file = <undefined>
subjects_dir = /home/bpinsard/softs/freesurfer/subjects
terminal_output = stream
tesselate_all_voxels = <undefined>
use_real_RAS_coordinates = <undefined>
, trait: subjects_dir, old value: <undefined>, new value: /home/bpinsard/softs/freesurfer/subjects
Traceback (most recent call last):
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 520, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 483, in dispatch
    handler( *args )
  File "nipype/interfaces/freesurfer/base.py", line 116, in _subjects_dir_update
    self.inputs.subjects_dir})
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3206, in update
    new_dic = self._validate_dic( dic )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3304, in _validate_dic
    raise excp
TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of u'SUBJECTS_DIR' <type 'unicode'> was specified.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "nipype/interfaces/freesurfer/base.py", line 110, in __init__
    self.inputs.subjects_dir = self._subjects_dir
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 475, in __call__
    self.notify_listener( self, object, trait_name, old, new )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 546, in _notify_method_listener
    listener )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 527, in _dispatch_change_event
    handle_exception( object, trait_name, old, new )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 520, in _dispatch_change_event
    self.dispatch( handler, *args )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_notifiers.py", line 483, in dispatch
    handler( *args )
  File "nipype/interfaces/freesurfer/base.py", line 116, in _subjects_dir_update
    self.inputs.subjects_dir})
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3206, in update
    new_dic = self._validate_dic( dic )
  File "/home/bpinsard/anaconda2/envs/preproc/lib/python2.7/site-packages/traits/trait_handlers.py", line 3304, in _validate_dic
    raise excp
traits.trait_errors.TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of u'SUBJECTS_DIR' <type 'unicode'> was specified.

looking at the output of nosetest . in nipype git clone on master, many interfaces yield the exact same error.

The plugins do not yield errors in nosetest because the environ is not an input, but is set when job is submitted, so I do not know if this is possible to write test that would work without SGE installed.

bpinsard · 2016-09-14T16:18:29Z

I think there is something really weird happening.
If I run this call in ipython, it crashes similarly (type unicode)

TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of u'SUBJECTS_DIR' <type 'unicode'> was specified.

Then I go into debug and run the exact same line that crashed, it crashes with (type str):

ipdb> self.inputs.environ.update({'SUBJECTS_DIR':self.inputs.subjects_dir})
*** TraitError: Each key of the 'environ' trait of a MRITessellateInputSpec instance must be an implementor of, or can be adapted to implement, newstr or None, but a value of 'SUBJECTS_DIR' <type 'str'> was specified.

When I import os an check type of os.environ values

ipdb> os.environ['DISPLAY'].__class__
<type 'str'>
ipdb> os.environ.keys()[0].__class__
<type 'str'>

So when it is in the nipype code running, all my strings (even the one written in nipype code, such as 'SUBJECTS_DIR', as well as os.environ) get converted to unicode, which is the problem when type checking is done in traits.

oesteban · 2016-09-14T16:27:53Z

Does it replicate if you do

python -c "from __future__ import unicode_literals; from nipype.pipeline import engine as pe; from nipype.interfaces import freesurfer; n_tesselate = pe.Node(freesurfer.MRITessellate(label_value=1), name='tesselate')"

?

Thanks

bpinsard · 2016-09-14T16:30:00Z

yes it replicate the bug

bpinsard · 2016-09-15T19:29:05Z

ok just to update on this issue, that is in fact not solved for me.
The sourced script in my bashrc was in fact Freesurfer setup script, that I of course need as most of us.

So as to try the exact line that fail for string being unicode I did a small script:
As the class was different in debug than in the running code in nipype

from __future__ import unicode_literals 
import os 

k = 'SUBJECTS_DIR'
print( k.__class__)
v = '/here'
print( v.__class__)

Which outputs:

$ python  test.py
<type 'unicode'>
<type 'str'>

When I remove the __future__ unicode_literals import

$ python  test.py 
<type 'str'>
<type 'str'>

So it seems that future converts strings defined in code in unicode.
Do you have the same output when you run this script?

How should I force this string to be newstr or newbytes?

What is the rationale of importing unicode_literals everywhere?
However the question which remains is : why am I the only one to have this bug?

Thanks

bpinsard · 2016-09-15T19:39:46Z

the above PR fixes the bug...

oesteban · 2016-09-15T19:39:59Z

The rationale behind importing unicode_literals is that it makes python 2 behave as python 3 as regards strings.

Regarding your examples, how is that possible that your in your first example you have two different types? I get unicode for both strings.

bpinsard · 2016-09-15T19:42:57Z

does os.environ includes future? I don't know.
But if you get unicode in both cases, how type checking in traits work for you with all the dictstrstr that requires newstr or newbytes??

oesteban · 2016-09-15T19:48:12Z

I mean:

from __future__ import unicode_literals 
import os 

k = 'SUBJECTS_DIR'
print( k.__class__)
v = '/here'
print( v.__class__)

Maybe you meant:

from __future__ import unicode_literals 
import os 

k = os.getenv('SUBJECTS_DIR')
print( k.__class__)
v = '/here'
print( v.__class__)

?

bpinsard · 2016-09-15T19:52:32Z

oups sorry, I pasted then updated my code, you got it correct it was

k = os.environ['SUBJECTS_DIR']

- Close nipy#1655: fixes the References:: leftover in help() of interfaces - Close nipy#1644: fixes the parameterized paths generated with python 2 - All uses of repr and re-definitions of __repr__ have been revised (again, this is related to nipy#1644, nipy#1621, etc).

bpinsard changed the title ~~python 2.7~~ python 2.7 string issue in plugins Sep 12, 2016

oesteban mentioned this issue Sep 12, 2016

[FIX] Encoding problems in SGE plugin #1624

Closed

oesteban added a commit to oesteban/nipype that referenced this issue Sep 12, 2016

[FIX, WIP] Encoding problems in SGE plugin

a570796

Fixes nipy#1621

oesteban mentioned this issue Sep 13, 2016

Fix dictstrstr #1629

Merged

bpinsard closed this as completed Sep 15, 2016

bpinsard reopened this Sep 15, 2016

bpinsard mentioned this issue Sep 15, 2016

fixing unicode_literals issues #1638

Closed

oesteban mentioned this issue Sep 21, 2016

[FIX] Several fixes related to unicode literals #1656

Merged

mgxd added the bug label Oct 12, 2016

mgxd closed this as completed Oct 12, 2016

djarecka mentioned this issue Jan 29, 2017

segfault running /pipeline/engine/tests/test_utils.py #1788

Closed

oesteban mentioned this issue Feb 11, 2017

[BUG] Need traits>=4.5: unicode string problems #1809

Closed

python 2.7 string issue in plugins #1621

python 2.7 string issue in plugins #1621

Comments

bpinsard commented Sep 12, 2016

oesteban commented Sep 12, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

oesteban commented Sep 13, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

oesteban commented Sep 13, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

chrisgorgo commented Sep 13, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

chrisgorgo commented Sep 13, 2016

Uh oh!

bpinsard commented Sep 13, 2016

Uh oh!

chrisgorgo commented Sep 13, 2016

Uh oh!

oesteban commented Sep 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oesteban commented Sep 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bpinsard commented Sep 14, 2016

Uh oh!

bpinsard commented Sep 14, 2016

Uh oh!

oesteban commented Sep 14, 2016

Uh oh!

bpinsard commented Sep 14, 2016

Uh oh!

bpinsard commented Sep 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bpinsard commented Sep 15, 2016

Uh oh!

oesteban commented Sep 15, 2016

Uh oh!

bpinsard commented Sep 15, 2016

Uh oh!

oesteban commented Sep 15, 2016

Uh oh!

bpinsard commented Sep 15, 2016

Uh oh!

oesteban commented Sep 13, 2016 •

edited

Loading

oesteban commented Sep 13, 2016 •

edited

Loading

bpinsard commented Sep 15, 2016 •

edited

Loading