Upgrading pip fails on Windows when install path is too long

[zooba]

Received the following log on microsoft/PTVS#782.

Virtual environment is being created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
Virtual environment was successfully created at 'C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env'
----- Installing 'pip' -----
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-7.1.2-py2.py3-none-any.whl#md5=5ff9fec0be479e4e36df467556deed4d
  Using cached pip-7.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 6.0.8
    Uninstalling pip-6.0.8:
      Exception:
      Traceback (most recent call last):
        File "C:\Python34\lib\shutil.py", line 523, in move
          os.rename(src, real_dst)
      FileNotFoundError: [WinError 3] The system cannot find the path specified: 'c:\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc' -> 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\basecommand.py", line 232, in main
          status = self.run(options, args)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\commands\install.py", line 347, in run
          root=options.root_path,
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_set.py", line 543, in install
          requirement.uninstall(auto_confirm=True)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_install.py", line 667, in uninstall
          paths_to_remove.remove(auto_confirm)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\req\req_uninstall.py", line 126, in remove
          renames(path, new_path)
        File "C:\Users\TrevorSullivan\Source\Repos\PythonApplication11\PythonApplication11\env\lib\site-packages\pip\utils\__init__.py", line 316, in renames
          shutil.move(old, new)
        File "C:\Python34\lib\shutil.py", line 535, in move
          copy2(src, real_dst)
        File "C:\Python34\lib\shutil.py", line 245, in copy2
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "C:\Python34\lib\shutil.py", line 109, in copyfile
          with open(dst, 'wb') as fdst:
      FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\TREVOR~1\\AppData\\Local\\Temp\\pip-h64zdfhc-uninstall\\users\\trevorsullivan\\source\\repos\\pythonapplication11\\pythonapplication11\\env\\lib\\site-packages\\pip\\_vendor\\requests\\packages\\urllib3\\packages\\ssl_match_hostname\\__pycache__\\_implementation.cpython-34.pyc'
----- Failed to install 'pip' -----

The problem seems to be that the entire install path is replicated beneath TEMP, which very quickly exceeds the maximum path length supported by Windows. I guess the aim is to be able to rollback a failed uninstall (which also fails here, and leaves corrupt state), but we may need an alternative to including the full path - maybe generate some sort of map file as well?

[xavfernandez]

Related to #2892

[RussBaz]

I am still experiencing the same issue even though I am using pip 7.1.2 and python 3.5.1.

Are there any updates available regarding this issue?

[brettcannon]

Just ran into this under pip 8.1.1 and Python 3.5.2 on Windows 10.

[GadgetSteve]

I am wondering if a work around for Windows users might not be to check for paths that are likely to exceed the limit, (260 chars total), and using subst to mount a drive at a suitable point, i.e. identify a spare drive letter and mount that letter to as the TEMP - of course this would not truly fix the problem and deeply nested packages could still hit problems - that is down to Microsoft actually addressing the MAXPATH issue once and for all. If the installation succeeds then the temporary area could be deleted and the subst deleted. In the case of a TEMP location if it fails then the path on the "new" drive could be used for recovery before deletion and the subst deleted. Of course it cannot help someone who already has 23 drive letters mapped A: B: & C: are reserved!

In the case of a virtual environment facing the issue I think that the best that can be done is to issue the user an error with a suggestion that they make the virtual environment with --relocatable and either move it higher up the directory tree or use subst to put it at the top of a "new" drive.

I have just done a quick test on a Windows 10 (Anniversary Update) 64 bit machine and it would quite happily allow me to create a directory 217 characters long, (at which point the next mkdir failed and then mount it with 'subst' to a new drive letter and create a path 117 characters long on that drive - all that I had time for.

[zooba]

FWIW, Windows 10 allows the option to disable MAX_PATH limits for "self-certified" applications, which will include Python 3.6. Currently there's a machine policy that needs to be enabled, and the Python 3.6 installer will prompt users to do that (if they can - need to be admin).

So the issue will go away in the future, and anything else we do now is mitigation for people on existing setups (which obviously has a lot of value, since it's going to be a long time before everyone is on Python 3.6 on Windows 10).

[GadgetSteve]

@zooba - My reading so far suggests that the disable MAX_PATH limits option is only planned to be available or possibly only accessible in Widows 10 Pro and Enterprise editions not the home edition.

[zooba]

@GadgetSteve Where are you reading that? It's highly unlikely that such core APIs would be different between Windows SKUs, though of course there's no support included in the home editions for managing group policy. Shouldn't prevent the registry edits from working though.

[GadgetSteve]

@zooba It looks like the only way to enable the change for the user is via the Group Policy Editor which is missing from the Home editions - I have had too many changes that should work via registry edits fail to work, revert on a MS Update or work erratically, (e.g. I cannot get the numb-lock to survive a reboot on Win 10 even after the registry edits), to be completely happy with the idea of relying on them.

[pradyunsg]

@pfmoore Any ideas on what could be a possible workaround for this?

[GadgetSteve]

On 21/08/2017 11:29, Pradyun Gedam wrote: @pfmoore <https://github.com/pfmoore> Any ideas on what could be a possible workaround for this?

If you are on Windows 10 applying the changes detailed at https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/ is a very good option. Otherwise options include: Installing python to a short path, e.g. C:\Py36 for Python 3.6 and setting temp to c:\tmp can sometimes help. If the path that is too long is the cashe path you can use `pip --cache-dir=C:\PyCashe` and the --root and --prefix options can also be helpful and sometimes you can make a long path into a short one with the command line subst and mklink commands. Of course there is always the option to switch to Linux... -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer.

…

--- This email has been checked for viruses by AVG. http://www.avg.com