fail to precompile

[huckl3b3rry87]

I am running into this precomilation error:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.1 (2017-03-05 13:25 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> using PyPlot
INFO: Installing matplotlib via the Conda matplotlib package...
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
matplotlib                2.0.0               np112py27_0  
ERROR: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name multiarray',)
  File "/usr/lib/python2.7/dist-packages/matplotlib/__init__.py", line 122, in <module>
    from matplotlib.cbook import is_string_like, mplDeprecation, dedent, get_label
  File "/usr/lib/python2.7/dist-packages/matplotlib/cbook.py", line 33, in <module>
    import numpy as np
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray

 in pyerr_check at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:56 [inlined]
 in pyerr_check at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:61 [inlined]
 in macro expansion at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:81 [inlined]
 in pyimport(::String) at /home/febbo/.julia/v0.5/PyCall/src/PyCall.jl:392
 in pyimport_conda(::String, ::String, ::String) at /home/febbo/.julia/v0.5/PyCall/src/PyCall.jl:485
 in __init__() at /home/febbo/.julia/v0.5/PyPlot/src/init.jl:177
 in _include_from_serialized(::String) at ./loading.jl:150
 in _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:187
 in _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:217
 in require(::Symbol) at ./loading.jl:371
during initialization of module PyPlot

Looking at the readme for pycall:

https://github.com/JuliaPy/PyCall.jl#using-pycall-from-julia-modules

It looks like we may be able to fix these issues with this strategy?

[stevengj]

Something seems weird with your Python configuration. It looks like you are using Conda.jl, but it is still trying to load things from /usr/lib/python2.7/.

If you look at your environment variables (ENV in Julia), do you have anything like PYTHONPATH or PYTHONHOME set? You could be running into JuliaPy/Conda.jl#69

[huckl3b3rry87]

@stevengj thank you for the help!

Looking at my python related ENV variables:

PYTHONPATH=/opt/ros/kinetic/lib/python2.7/dist-packages:/usr/lib/python2.7/dist-packages

Are you suggesting that I somehow remove PYTHONPATH? I think this got messed up when I started using RobotOS. Is that possible? I don't want to mess that environmental variables for that package up, if this may be the case, is there a way to run both packages without error?

Thanks!

[huckl3b3rry87]

if I just do a

using Conda

that fixes it

[huckl3b3rry87]

So, when I use PyPlot. jl in conjunction with Plots.jl, this patch fails, so I tried:

julia> using PyCall

julia> Pkg.build("PyCall")
INFO: Building Conda
INFO: Building PyCall
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/febbo/.julia/v0.5/Conda/deps/usr:

The following packages will be UPDATED:

    numpy: 1.12.0-py27_0 --> 1.12.1-py27_0

numpy-1.12.1-p 100% |############################################################################################################################################################| Time: 0:00:00  11.48 MB/s
INFO: PyCall is using /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python (Python 2.7.13) at /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python, libpython = /home/febbo/.julia/v0.5/Conda/deps/usr/lib/libpython2.7
INFO: /home/febbo/.julia/v0.5/PyCall/deps/deps.jl has not changed
INFO: /home/febbo/.julia/v0.5/PyCall/deps/PYTHON has not changed

julia> import Conda

julia> Conda.PYTHONDIR
"/home/febbo/.julia/v0.5/Conda/deps/usr/bin"

julia> ENV["PYTHON"] = "/home/febbo/.julia/v0.5/Conda/deps/usr/bin"
"/home/febbo/.julia/v0.5/Conda/deps/usr/bin"

julia> Pkg.build("PyCall")
INFO: Building Conda
INFO: Building PyCall
INFO: No system-wide Python was found; got the following error:
could not spawn `/home/febbo/.julia/v0.5/Conda/deps/usr/bin -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))"`: permission denied (EACCES)
using the Python distribution in the Conda package
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
numpy                     1.12.1                   py27_0  
INFO: PyCall is using /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python (Python 2.7.13) at /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python, libpython = /home/febbo/.julia/v0.5/Conda/deps/usr/lib/libpython2.7
INFO: /home/febbo/.julia/v0.5/PyCall/deps/deps.jl has not changed
INFO: /home/febbo/.julia/v0.5/PyCall/deps/PYTHON has not changed

julia> using PyPlot

julia> Pkg.build("Plots")
INFO: Building Plots

julia> Pkg.build("PyPlot")
INFO: Building Conda
INFO: Building PyCall
INFO: No system-wide Python was found; got the following error:
could not spawn `/home/febbo/.julia/v0.5/Conda/deps/usr/bin -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))"`: permission denied (EACCES)
using the Python distribution in the Conda package
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
numpy                     1.12.1                   py27_0  
INFO: PyCall is using /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python (Python 2.7.13) at /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python, libpython = /home/febbo/.julia/v0.5/Conda/deps/usr/lib/libpython2.7
INFO: /home/febbo/.julia/v0.5/PyCall/deps/deps.jl has not changed
INFO: /home/febbo/.julia/v0.5/PyCall/deps/PYTHON has not changed

julia> using PyPlot

julia> 

But then when I start a new terminal I get the error that I started this post with.

[huckl3b3rry87]

@stevengj

So, if I follow the instructions and delete:

  export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages

in my .bashrc then it works.

Is there a way that I can still have this for other software and still use PyPlot?

[stevengj]

@huckl3b3rry87 now that you've built PyCall and PyPlot, I think it should work in future imports even if you don't unset the environment variables?

In any case, in the the next release of Conda, the issue should be fixed: JuliaPy/Conda.jl#70 ... you can do Pkg.checkout("Conda") to get the latest version now.

[stevengj]

I've also just pushed a patch to PyCall that should make its build script more resilient to the environment.

[stevengj]

Can you verify that with the latest PyCall and Conda (Pkg.checkout both of them), the problem goes away even if you re-build PyCall and PyPlot with your PYTHONPATH set?

[huckl3b3rry87]

@stevengj thanks for helping with this!

Unfortunately, it does not see, to work:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.1 (2017-03-05 13:25 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> Pkg.checkout("PyCall")
INFO: Checking out PyCall master...
INFO: Pulling PyCall latest master...
INFO: No packages to install, update or remove

julia> Pkg.checkout("PyPlot")
INFO: Checking out PyPlot master...
INFO: Pulling PyPlot latest master...
INFO: No packages to install, update or remove

julia> Pkg.build("PyCall")
INFO: Building Conda
INFO: Building PyCall
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
numpy                     1.12.1                   py27_0  
INFO: PyCall is using /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python (Python 2.7.13) at /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python, libpython = /home/febbo/.julia/v0.5/Conda/deps/usr/lib/libpython2.7
INFO: /home/febbo/.julia/v0.5/PyCall/deps/deps.jl has not changed
INFO: /home/febbo/.julia/v0.5/PyCall/deps/PYTHON has not changed

julia> Pkg.build("PyPlot")
INFO: Building Conda
INFO: Building PyCall
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
numpy                     1.12.1                   py27_0  
INFO: PyCall is using /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python (Python 2.7.13) at /home/febbo/.julia/v0.5/Conda/deps/usr/bin/python, libpython = /home/febbo/.julia/v0.5/Conda/deps/usr/lib/libpython2.7
INFO: /home/febbo/.julia/v0.5/PyCall/deps/deps.jl has not changed
INFO: /home/febbo/.julia/v0.5/PyCall/deps/PYTHON has not changed

julia> Pkg.checkout("Conda")
INFO: Checking out Conda master...
INFO: Pulling Conda latest master...
INFO: No packages to install, update or remove

julia> Pkg.build("Conda")
INFO: Building Conda

Then add this to $HOME\.bashrc (i.e. not /etc/bash.bashrc):

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages

Then:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.1 (2017-03-05 13:25 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> using PyPlot
INFO: Recompiling stale cache file /home/febbo/.julia/lib/v0.5/Conda.ji for module Conda.
INFO: Recompiling stale cache file /home/febbo/.julia/lib/v0.5/PyPlot.ji for module PyPlot.
INFO: Installing matplotlib via the Conda matplotlib package...
Fetching package metadata .........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/febbo/.julia/v0.5/Conda/deps/usr:
#
matplotlib                2.0.0               np112py27_0  
ERROR: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name multiarray',)
  File "/usr/lib/python2.7/dist-packages/matplotlib/__init__.py", line 122, in <module>
    from matplotlib.cbook import is_string_like, mplDeprecation, dedent, get_label
  File "/usr/lib/python2.7/dist-packages/matplotlib/cbook.py", line 33, in <module>
    import numpy as np
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray

 in pyerr_check at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:56 [inlined]
 in pyerr_check at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:61 [inlined]
 in macro expansion at /home/febbo/.julia/v0.5/PyCall/src/exception.jl:81 [inlined]
 in pyimport(::String) at /home/febbo/.julia/v0.5/PyCall/src/PyCall.jl:392
 in pyimport_conda(::String, ::String, ::String) at /home/febbo/.julia/v0.5/PyCall/src/PyCall.jl:485
 in __init__() at /home/febbo/.julia/v0.5/PyPlot/src/init.jl:172
 in _include_from_serialized(::String) at ./loading.jl:150
 in _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:187
 in require(::Symbol) at ./loading.jl:394
during initialization of module PyPlot

julia> Pkg.status("Conda")
 - Conda                         0.5.1+             master

julia> Pkg.status("PyPlot")
 - PyPlot                        2.3.1+             master

julia> Pkg.status("PyCall")
 - PyCall                        1.11.0+            master

julia> ENV["PYTHON"]
ERROR: KeyError: key "PYTHON" not found
 in access_env(::Base.##351#352, ::String) at ./env.jl:45
 in getindex(::Base.EnvHash, ::String) at ./env.jl:73

[stevengj]

You need Pkg.checkout("Conda") too.

[huckl3b3rry87]

@stevengj I did that. You can see that I am on the master branch

[stevengj]

I'm having trouble reproducing your problem; I set PYTHONHOME to an alternate python's location and it still works fine with Conda.

[huckl3b3rry87]

I am getting the error with PYTHONPATH

[stevengj]

Okay, I got an error if I set PYTHONPATH, not PYTHONHOME.

I think the basic problem here is that PYTHONPATH is used by libpython at runtime to populate sys.path. We could ignore PYTHONPATH in PyCall (e.g. by temporarily unsetting it while python initializes), but that seems wrong: PYTHONPATH is supposed to be something that users can do to control the search path of Python.

I'm not sure if this is fixable in a reasonable way.

If you need to set the python path specifically for one python executable, using an environment variable is the wrong way — instead, you create a file /usr/lib/python2.7/site-packages/mypath.pth with a line containing the directory to add.

[huckl3b3rry87]

@stevengj thank you very much for helping with this!

I will do what you suggested:)