Parallelizing the install process + PoC!

[bmartinn]

First of all, kudos guys on the v20+ release 👍 , wheel caching is much better and the addition of cached built wheels is an awesome time saver.

TL;DR
Reference implementation for pip install --parallel , which will download inspect and install packages in parallel, highly accelerating the installation environment of an entire project (or actually any requirements.txt file).
A few speedup numbers (Ubuntu w/ python 3.6):

Requirements variant  | Parallel+D/L | PyPi v20.1+D/L | Speedup incl' Download | Parallel | PyPi v20.1 | Speedup Cached Wheels
--------------------- | ------------ | -------------- | ---------------------- | -------- | ---------- | ----------------------
A                     | 0:58.14      | 1:51.08        | x1.90                  | 0:16.16  | 0:33.57    | x2.08
B                     | 5:47.36      | 6:55.49        | x1.19                  | 0:52.19  | 1:12.07    | x1.37
C                     | 0:56.08      | 1:44.34        | x1.86                  | 0:14.36  | 0:29.21    | x2.01
D                     | 0:36.45      | 1:39.55        | x2.71                  | 0:14.59  | 0:33.20    | x2.22
--------------------- | ------------ | -------------- | ---------------------- | -------- | ---------- | ----------------------
Average               |              |                | x1.91                  |          |            | x1.92

Details:
We heavily rely on pip to set up environments on our ever changing production systems.
Specifically either setting a new virtual environment or inside a docker, the full implementation of our ML/DL agent is open-source and can be found here: https://github.com/allegroai/trains

With the introduction of pip 20.1 we decided to try and parallelize the pip install process in order to save precious spin up time.

1. The package resolving & download was moved into a ThreadPool (RequirementSet was added a global lock). The only caveat is the download progress bar (more on that down below).

The reason for using Threads is the fact that most of the speedup here is Network and I/O which are parallelized well on python Threads. The second reason is the way RequirementSet is constancy growing with every package discovered, and it's requirements. Sharing the set among all threads
is quite trivial, as opposed to sharing them across processes.

2. Wheel files (only) installation process moved to a Process Pool, as installing the wheels will not execute any script, and can be parallelized without risk.

The reason for choosing Processes over Threads is the limiting factor here is actually CPU cycles in the installation process. Also the Wheel unpacking and copying is totally independent from one another and the rest of the process, and lastly order has no meaning in this stage as the requirements order is only important when building packages.

3. Unpacking the wheels was optimized to reuse the unpacked folders, this happens between the inspection part (resolver) on a cached wheel and the installation of the same wheel (essentially what happened was the wheel was unpacked twice)

4. Solving the parallel progress bar: The progress bar has now a global lock, the first progress-bar to acquire it, will be the one outputting the progress to the tty. Once the progress-bar is done, another instance can acquire the lock and will continue to report from its current progress stage.
   This looks something like:

|████████████████████████████████| 101 kB 165 kB/s 
|█                           | 71 kB 881 kB/s eta 0:00:03

The full reference code can be found here
Usage example:

pip install --parallel -r requirements.txt

[bmartinn]

Benchmark test details.

Take the download (D/L) column time/speed with a grain of salt, download speed is not stable enough between runs, but obviously parallel download is much more efficient.

Requirements variant B is actually somewhat of an outlier, it contains the torch==1.5.0 package, which is a 700+Mb package. Obviously the download and the installation process is limited by the single slowest package, and that is the case for variant B. The reason for having it in the first place, is that we use it quite often (and also TensorFlow, a package almost the same size). For comparison scikit-learn is only a few tens of Mb.
The other variants are a more off-the-shelf mixture of packages.

Notice that in requirements variants A/B/C all the package versions are fixed.
Attached requirement files used for the benchmark:

reqsA.txt
reqsB.txt
reqsC.txt
reqsD.txt

[McSinyx]

Thank you for sharing the benchmark, the design as well as the proof of concept. I am glad to see the positive benchmark for larger (I supposed by estimating the size of others based on that of B, I'd love to have them to test in the future though) collections, compared to the ones tested under GH-3981.

I made a quick run and found that the PoC still have the issue with

We need to find a way to clean up correctly when SIGINT is issued. Since the progress bar is initialized in a different thread we cannot set the signal handler as InterruptibleMixin did before. That functionality is currently commented out and needs to be somehow restored before we merge this.

I am unaware if this is because you and your team has not worked on that or there's still a technical obstacle to be solved. If the latter is the case, I'd love to know what you found in extra to the quote above.

I've also submitted a plan for solving GH-825 as part of GSoC, however it's purely a plan. Theoretically it should work, but I can't be sure that it definitely will in practice. I'd be really grateful if you take a look at it and leave some comments. Note that I'll only know if it's approved by pip's maintainers tomorrow (May 5), so don't take it too seriously.

Given the situation in GH-8169, I wonder how Pool performs under a single-core machine, and is there any perk that we're yet to be aware of. I also suspect multiprocessing.ThreadPool to have same problem with multiprocessing.dummy.Pool, however I'm not sure on how to reproduce (that's why the issue was opened in the first place).

[bmartinn]

Thanks @McSinyx for the references!

To answer a few of your concerns:

1. have them to test in the future though) compared to the ones tested under Download requirements in parallel #3981.

Requirement files added here

2. Notice that Download requirements in parallel #3981 implementation is quite outdated (more than 3 years old), and actually the good people of pypa did a great a job with v20, simplifying the process of package download and dependency resolving. This is the main reason for us to tackle this issue now, as the solution is relatively straight forward.

3. If you have a single core, Pool should not be used (performance loss is negligible, but why would you create another subprocess, right :).
   See the if statement and program flow here
   You could actually add to the if multiprocessing.cpu_count() > 1 and that will make sure that even if one uses the --parallel flag but has a single core, there is no performance loss.
   The same can be added here

4. Regrading the Progress Bar signal handler, see here
   Basically only the main thread will replace the handler, any other thread will just ignore it.
   This means that ctrl-c will be caught and processed, while still allowing multiple threads to report progress bars. To be honest, the self.finish() that is called, on the signal handler seems quite redundant, the only thing it does is flush the tty, and return the cursor. Both will happen anyhow when the process quits (just my 2 cents :)...

On a different note, if you are willing to help out with this code and make it PR worthy, please don't be a stranger, feel free to push to the forked repo 😄

[raviv]

This is awesome.
Looking forward for the PR.

[uranusjr]

Also note that pip is in the middle of the process of integrating a new resolver. So make sure you're able to also introduce the parallelism to the new resolver, otherwise all the good work would be lost once the transition is finished.

[bmartinn]

@uranusjr thanks for the heads up!
Where could I find the new resolver ?

EDIT:
Are you referring to ?
https://github.com/pypa/pip/blob/master/src/pip/_internal/resolution/resolvelib/resolver.py
If so, the implementation is almost the same, should work without any real effort.
How do I make sure I'm using it (seems like the default is "legacy")?

[uranusjr]

@bmartinn Your observations are correct. The new resolver can be enabled (in 20.1 or later) by passing --unstable-feature=resolver to pip.

[McSinyx]

@bmartinn, thank you for the requirement files as well as explanations 2 and 3. As of the termination, yesterday I notice that after canceling in the middle of parallel download, my cursor disappeared, so there might be something wrong. However, to make the UI user-friendly, I believe that it might be a good idea to only having the main thead to take care of all of the displaying, as in my proposal, which is now accepted (yay, and thank PyPA maintainers and others for commenting during the drafting process).

To @uranusjr as well, I'm not sure how the proposal was reviewed, but if you haven't taken a look at it, could you please scheme through the implementation idea (section 3.3.3) if that fits your vision of coordinating into the new resolver, i.e. does that cause any trouble for the on going implementation of the resolver?

BTW I feel the need of saying that I've just got (physically) back to school this week and is still trying to get used to the new and denser schedule, so please tolerate me if in the next one or two days I am not able to carefully examine things to meaningfully participate in the discussion.

[bmartinn]

@McSinyx, following your suggestion, I pushed a fix for the Logger handler issue (basically forcing at least one instance to be created on the main thread), let me know if it solved the cursor problem.

@uranusjr I'm testing the --unstable-feature=resolver feature, and it seems to download tar.gz and not .whl files, are you guys switching to tag.gz ?

[uranusjr]

@bmartinn pip maintains a wheel cache and won’t redownload if you had downloaded a wheel previously. You probably need to supply --no-cache or something.

[bmartinn]

Apologies @uranusjr I was seeing the future download being tar.gz and apparently there is no wheel for it (it compiles), other packages behave as expected and download .whl files :)
Thanks!

[bmartinn]

Hi @uranusjr
I updated the code here to support the new Resolver as well as the legacy one. I like the new resolver / resolution design, and I tried to make the parallelization as agnostic as possible, basically there is another resolution resolver running in the background try to guess ahead what will be the package to download.

This design should hold with the continuous development of the new Resolver/Resolution classes.

With your approval I'd like to push it as a PR :)

BTW:
It seems the new resolver is slower then the current default one, but I guess that's reasonable as it looks like it is still under development 👍
Currently with the new experimental Resolver, we still get a x1.7 factor speedup through the parallelization, so at least we have that.

EDIT:
@uranusjr
Do you think it is better to have the ResolverLib parallelization PR to resolvelib or here?

[uranusjr]

I think it would be a reasonable addition, but some considerations are needed. threading (and by extension parts of multiprocessing) is not a universally available module (#8169), so a fallback to single-thread operation would be needed on such platforms. The main task (IMO) would be to come up with a reasonable abstraction so we don’t end up maintaing two partially duplicated pieces of code.