Clarify & document our CI strategy

[xavfernandez]

Of our combination of supported interpreters, OS and architecture we are currently only testing a few without clear strategy (cf https://github.com/pypa/pip/pull/7247/files)

The goal would be to come up with a bunch of rules like:

• we only test the latest X minor PyPy versions
• we have at least one full test (with integration tests) on x86 for each OS (except MacOS where it doesn't make sense anymore)
• we test at least one PyPy version on each OS (and not only Linux)
• etc.

[chrahunt]

We also need to consider:

• integration vs unit
• venv vs virtualenv (as mentioned in #7270 (comment))

[brainwane]

@ewdurbin and @pradyunsg are going to be talking about this issue this week as part of the donor-funded pip work that we want to complete by about the end of May.

[pradyunsg]

Based on a bunch of discussions lately, I think that we seem to have concensus to:

• Better document what resources we have from the various providers, so that it's easier to understand what could be a bottleneck + have a single place to find these details.
• Improve our testing code:
  • Improve error reporting in CI
  • Make it faster! (Speeding up Tests #4497)

[chrahunt]

Concretely for the "make it faster" task, I can walk someone through getting performance data on Windows and analyzing it like I had to for #7263. If we refine the tools and process then we'll truly have a lot of visibility into where time is being spent.

[pradyunsg]

@chrahunt 🙋🏻‍♂️

This sounds like it would also be a good process to document in some form - either in pip's documentation if it is very-pip-specific or as a blog post somewhere, for broader visibility.

[duckinator]

Adjacent to making the entire test suite faster, there's also the concept of "failing fast." Basically, you want run the tests which are more likely to fail first. This is usually tests related to code that was changed since the last test run.

With regards to that, it may be worth looking into using something along the lines testmon. I've not worked with testmon specifically, but I've heard good things about both the tool and the general approach it takes.

As a note, a prerequisite of this is to know that the tests aren't order-dependent. So it may be worth trying to use something like pytest-randomly (where it can be used, since it's only Python 3.5+) first, and resolve any problems that show up there.

[pradyunsg]

With regards to that, it may be worth looking into using something along the lines testmon. I've not worked with testmon specifically, but I've heard good things about both the tool and the general approach it takes.

The tricky thing with this, is that our tests invoke a subprocess of pip, which isn't very friendly for tools like this.

As a note, a prerequisite of this is to know that the tests aren't order-dependent.

We parallelize and isolate our tests extensively, so they're definitely not order-dependent. :)