jsonschema and vcversioner

[onethirtyfive]

Hi there,

This is more of a request for "what do I do?" than an issue with pypi2nix.

i want to use "jsonapi-client", which in turn depends on "jsonschema" and "async_timeout", which in turn depends on "vcversioner" as a dependency. Here's some context and related issues:

https://pypi.python.org/pypi/nexus_uploader/1.0.10 (ctrl+f "pip install - Download error")

And the author of jsonschema saying something about "install_requires" and this not being an issue:
python-jsonschema/jsonschema#276

I get a near-identical stack track as the one in jsonschema/#276 issue when I start a shell using only the pypi2nix-generated requirements.nix for these requirements:

requests==2.18.4
jsonapi-client==0.9.6

I could supposedly solve this by doing a pip install vcversioner "beforehand," but I don't know what "before" generating my requirements.nix means here.

Here is my exact output from nix-shell:

[snip]
creating build/bdist.linux-x86_64/wheel/async_timeout-2.0.0.dist-info/WHEEL
creating '/tmp/nix-build-python3.6-async-timeout-2.0.0.drv-0/async-timeout-2.0.0/dist/async_timeout-2.0.0-py3-none-any.whl' and adding '.' to it
adding 'async_timeout/__init__.py'
adding 'async_timeout-2.0.0.dist-info/DESCRIPTION.rst'
adding 'async_timeout-2.0.0.dist-info/metadata.json'
adding 'async_timeout-2.0.0.dist-info/top_level.txt'
adding 'async_timeout-2.0.0.dist-info/WHEEL'
adding 'async_timeout-2.0.0.dist-info/METADATA'
adding 'async_timeout-2.0.0.dist-info/RECORD'
Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found!
Couldn't find index page for 'vcversioner' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
No local packages or working download links found for vcversioner>=2.16.0.0
Traceback (most recent call last):
  File "nix_run_setup.py", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 42, in <module>
    vcversioner={"version_module_paths" : ["jsonschema/_version.py"]},
  File "/nix/store/839vm39v1wnb6zw0hn66g834n08735yg-python3-3.6.3/lib/python3.6/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/setuptools/dist.py", line 335, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/setuptools/dist.py", line 456, in fetch_build_eggs
    replace_conflicting=True,
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/pkg_resources/__init__.py", line 863, in resolve
    replace_conflicting=replace_conflicting
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1141, in best_match
    return self.obtain(req, installer)
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1153, in obtain
    return installer(requirement)
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/setuptools/dist.py", line 522, in fetch_build_egg
    return cmd.easy_install(req)
  File "/nix/store/lwjzzihibhqq3qangp8s68xkv7qxf2qi-python3.6-bootstrapped-pip-9.0.1/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 666, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('vcversioner>=2.16.0.0')
installing
/tmp/nix-build-python3.6-async-timeout-2.0.0.drv-0/async-timeout-2.0.0/dist /tmp/nix-build-python3.6-async-timeout-2.0.0.drv-0/async-timeout-2.0.0
running bdist_wheel
builder for ‘/nix/store/q033bysgbr9qdj1q004hswxr3fphhz5k-python3.6-jsonschema-2.6.0.drv’ failed with exit code 1
error: build of ‘/nix/store/q033bysgbr9qdj1q004hswxr3fphhz5k-python3.6-jsonschema-2.6.0.drv’ failed
/run/current-system/sw/bin/nix-shell: failed to build all dependencies

I'm not very proficient with python packaging. I hope there's an obvious solution.

Thanks for your time.

[onethirtyfive]

I should add that I had to move to nixos-unstable (18.03-pre), so that may have somehting to do with it. (I hope not.)

edit: nope, rolled back to 17.03 and same error.

[onethirtyfive]

So I learned today that the old setup_requires stuff can really give you a headache. I fixed my own issue with in requirement_overrides.nix:

{ pkgs, python }:

self: super: {
  "jsonschema" = python.overrideDerivation super."jsonschema" (old: {
    preConfigure = ''
      sed -i -e "s|setup_requires=\[.*\],|setup_requires=\[\],|" setup.py  
    '';
  });
}

There were two competing libraries (jsonapi-requests and jsonapi-client) which I wanted to investigate, but in trying to set up jsonapi-requests, I would need to override setup_requires on both the library AND its deps, and it still didn't work. (There was a dep called tenacity using "pbr" Python Build Reliability, which when I disabled... just didn't package right at all.)

Is Python packaging really this insane? Thank you SO MUCH for making this tool to mitigate the insanity. Without it, nix wouldn't have anything to do with this crap. ;)

Closing this issue--maybe someone else will get use out of it.

[garbas]

@onethirtyfive tnx for fixing your own issue :) sadly we don't (yet) have a good fix for setup_requires stuff.

I am now thinking we should actually include your patch for every python packages, since in 99% it is exactly what you want to do. what do you think @blueyed, should we add this?

[onethirtyfive]

Just a thought. One of the deps which I seded setup_requires out of then used an optional parameter defined by one of the things it was setup_requiring later in setup.py.

Take a look here: https://github.com/jd/tenacity/blob/master/setup.py#L18-L20

Merely editing out the setup_requires part of things will cause an indeterminate number of problems with all the libraries out there. It might be better to encourage a directory of custom overrides. (wiki?)

Some libraries like tenacity won't play well with pypi2nix because they depend on these setup_requires to work at all.

Just my 2c.