release tarballs on GitHub missing typeshed submodule, 0.470 source package missing from PyPI

[mr-c]

Hello,

pip download --no-binary :all: --no-deps mypy

only downloads the source package for version 0.256 (from the old mypy package)

and

pip download --no-binary :all: --no-deps mypy-lang

only downloads the source package for version 0.4.6

This made the latest release invisible to Debian via manual downloads or on our PyPI redirector http://pypi.debian.net/mypy

When switching to the GitHub release I failed to notice that the git submodule for the typeshed wasn't included: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=852819

Can the source package for 0.470 be uploaded to PyPI and/or the GitHub release tarball be updated? It doesn't make sense to offer them without the submodules included.

Thanks!

[JukkaL]

@gvanrossum Should we release PyPI source packages for future releases? Installing 0.256 is particularly troublesome. I assume that uploading source packages would be fairly simple?

[gvanrossum]

Sorry, I have absolutely no understanding of what those `pip` flags mean or how pip decides which file to download and what to do with it. It's a black box to me. Is a source pacakge whatever gets produced by `setup.py sdist`? That should be easy to add using twine, I suppose.

[mr-c]

Hey @gvanrossum , yep the setup.py sdist output needs uploading :-)

[gvanrossum]

OK, I've done that now. Let me know if there is more to be done.

[mr-c]

@gvanrossum confirmed. Is this now part of the release process?

[gvanrossum]

Hm, we really need to write down the steps for the release process in a file contained in the project.

[mr-c]

@gvanrossum I take it back, looks like the uploaded source package was made from an checkout of mypy without the submodules -- so the typeshed directory is still missing which puts us back with the same problem I started :-P

[gvanrossum]

No, I did a clean checkout followed by git submodule init and git submodule update, then I ran python3 setup.py dist.

I don't recall changing any of that. Does mypy-lang 0.4.6 have the same problem?

[gvanrossum]

The big change to setup.py was switching from default distutils to setuptools. Maybe the change is a side-effect from that? I've verified that the global variable data_files is filled with the typeshed files, but somehow it is being ignored by setup()...

[ambv]

Guido, I cannot reproduce your problem neither on master nor on the v0.470 tag. The distribution I am producing does contain everything, checked by unpacking the .tar.gz in dist/ (see here for my output).

Sadly, we cannot re-upload the .tar.gz for this version so I deleted the broken one and re-uploaded a .zip file instead (for some reason .tar.bz2 and .tar.xz are no longer allowed, possibly @dstufft knows why).

I can confirm this installs a working mypy from the .zip by doing this:

$ pip install --no-binary :all: mypy
Collecting mypy
  Using cached mypy-0.470.zip
Collecting typed-ast>=0.6.3 (from mypy)
  Using cached typed-ast-0.6.3.tar.gz
Installing collected packages: typed-ast, mypy
  Running setup.py install for typed-ast ... done
  Running setup.py install for mypy ... done
Successfully installed mypy-0.470 typed-ast-0.6.3

It works, I checked a few random files. I can confirm typeshed is installed successfully:

$ tree -L 3 lib
lib
├── mypy
│   ├── typeshed
│   │   ├── stdlib
│   │   ├── tests
│   │   └── third_party
│   └── xml
│       ├── mypy-html.css
│       ├── mypy-html.xslt
│       ├── mypy-txt.xslt
│       └── mypy.xsd
└── python3.6
    └── site-packages
        ├── __pycache__
        ├── _ast27.cpython-36m-darwin.so
        ├── _ast35.cpython-36m-darwin.so
        ├── easy_install.py
        ├── mypy
        ├── mypy-0.470-py3.6.egg-info
        ├── pip
        ├── pip-9.0.1.dist-info
        ├── pkg_resources
        ├── setuptools
        ├── setuptools-28.8.0.dist-info
        ├── typed_ast
        └── typed_ast-0.6.3-py3.6.egg-info

[mr-c]

Thank you @ambv and @gvanrossum for working on this. I can also confirm that the typeshed is now present in the source package.

[gvanrossum]

Thanks Lukasz. What a strange mystery. I suppose there might be different versions of setuptools or Python be involved? I will try to experiment.

[dstufft]

@ambv See PEP 527.

[gvanrossum]

I think I have a clue. I ran pip3 install -U setuptools and it gave this output:

Collecting setuptools
  Using cached setuptools-34.1.0-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools)
  Using cached packaging-16.8-py2.py3-none-any.whl
Requirement already up-to-date: six>=1.6.0 in /Users/guido/v36/lib/python3.6/site-packages (from setuptools)
Collecting appdirs>=1.4.0 (from setuptools)
  Using cached appdirs-1.4.0-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools)
  Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Installing collected packages: pyparsing, packaging, appdirs, setuptools
  Found existing installation: setuptools 29.0.1
    Uninstalling setuptools-29.0.1:
      Successfully uninstalled setuptools-29.0.1
Successfully installed appdirs-1.4.0 packaging-16.8 pyparsing-2.1.10 setuptools-34.1.0

After that, python3 setup.py sdist constructed a .tar.tz file containing typeshed/, runtests.py, test-data/, and mypy/test/. So I blame everything on an old version (29.0.1) of setuptools, or perhaps (the previous lack of) appdirs or packaging.

Gosh I hate dealing with this ecosystem.