list --outdated: TypeError: '>' not supported between instances of 'Version' and 'Version'

[daavve]

Environment

• pip version:
• Python version:
• OS:

Arch Linux, Testing version of pip: https://www.archlinux.org/packages/testing/any/python-pip/
Description

$ pip list -o

Exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
status = self.run(options, args)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run
packages = self.get_outdated(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 156, in get_outdated
dist for dist in self.iter_packages_latest_infos(packages, options)
File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 157, in
if dist.latest_version > dist.parsed_version
TypeError: '>' not supported between instances of 'Version' and 'Version'

[daavve]

Reported downstream: https://bugs.archlinux.org/index.php?do=details&task_id=58670

[yan12125]

Looks like an issue related to deblundling. If I modify the get_outdated() function in /usr/lib/python3.6/site-packages/pip/_internal/commands/list.py to:

    def get_outdated(self, packages, options):
        ret = []
        for dist in self.iter_packages_latest_infos(packages, options):
            print(dist.latest_version.__class__)
            print(dist.parsed_version.__class__)
            if dist.latest_version > dist.parsed_version:
                ret.append(dist)
        return ret

pip list -o prints:

<class 'pip._vendor.packaging.version.Version'>
<class 'pkg_resources.extern.packaging.version.Version'>
Exception:
Traceback (most recent call last):                                                                                                      
  File "/usr/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main                                               
    status = self.run(options, args)                                                                                                    
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 145, in run                                              
    packages = self.get_outdated(packages, options)                                                                                     
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/list.py", line 159, in get_outdated                                     
    if dist.latest_version > dist.parsed_version:                                                                                       
TypeError: '>' not supported between instances of 'Version' and 'Version'                                                               

pip._vendor.packaging.version.Version and pkg_resources.extern.packaging.version.Version are actually the same class (both are packaging.version.Version), while Python does not think so.

[pradyunsg]

Thanks for looking into this @yan12125! :)

Indeed, this is a downstream packaging issue.

[pradyunsg]

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

[daavve]

I think you are right. This looks like an Arch issue.

[yan12125]

I managed to patch pkg_resources locally to make pip list -o work with Arch Linux packages. Hopefully that helps other distributions using debundling. This patch is created against python-setuptools 1:39.2.0-1.

--- pkg_resources/__init__.py.orig	2018-05-22 00:21:05.592049050 +0800
+++ pkg_resources/__init__.py	2018-05-22 00:21:19.378315213 +0800
@@ -71,11 +71,11 @@
 
 from . import py31compat
 from pkg_resources.extern import appdirs
-from pkg_resources.extern import packaging
-__import__('pkg_resources.extern.packaging.version')
-__import__('pkg_resources.extern.packaging.specifiers')
-__import__('pkg_resources.extern.packaging.requirements')
-__import__('pkg_resources.extern.packaging.markers')
+import packaging
+__import__('packaging.version')
+__import__('packaging.specifiers')
+__import__('packaging.requirements')
+__import__('packaging.markers')
 
 
 if (3, 0) < sys.version_info < (3, 3):

[eli-schwartz]

Adding to my above comment, I don't see anything actionable on this from pip's on this -- it's due to how Arch Linux is debundling pip.

We're debundling according to the method nominally supported by pip itself, so I think it should be actionable here. :)

pip._internal.index.PackageFinder attempts to use pip._vendor.packaging.version.parse (which is dist.latest_version in the traceback), then compares it to pip._internal.utils.misc.get_installed_distributions which uses pip._vendor.pkg_resources to return a set of pkg_resources packages -- even though they're the same files, they appear in different locations on the import path, and python complains.

This somehow works for pip when using vendored everything, but only because the vendored pkg_resources is then additionally modified from stock pkg_resources to import everything from the top-level _vendor.packaging

...

I don't fully grok the magic used by pip's vendored() and pkg_resource's VendorImporter(), but it looks like they might be incompatible, and pip is papering over this logic by completely ignoring the latter in its own vendoring code.

[pradyunsg]

Hey @eli-schwartz!

Could you take a look at #5346 (comment) and provide information about the people at Arch Linux to ping for such things? :)

(I can't respond to your comment now because I'm super short on time; I'll reopen this issue so that I remember that I need to get back to this.)