Investigation: Why did Windows get 10% slower since 08-17?

[mdboom]

Strangely, the thrift benchmark got way faster (between 12x - 15x faster), while most other benchmarks got slower on the order of up to 30%. This seems like a bug or massive change in the thrift benchmark -- it is not just a one-off problem with either the head or base measurement, however, as we see it with both 64-bit and 32-bit Windows (but not other platforms). The thrift benchmark itself and its dependencies has not been updated during that time.

As a first step, I am re-running 08/24 and 08/17 to rule out anything that may have changed on the machine, external to the CPython source code (and will report back).

Relevant pages:

• 08/24: https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20240824-3.14.0a0-52caaef
• 08/17: https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20240817-3.14.0a0-79c542b

[itamaro]

could be related to python/cpython#122975 that was merged within that time window?
while it's about the free-threaded build, maybe it also affects the regular build?

[mdboom]

could be related to python/cpython#122975 that was merged within that time window? while it's about the free-threaded build, maybe it also affects the regular build?

Seems worth measuring. We measured its effect on a free-threaded build and it got slightly faster, but maybe not on a default build.

[itamaro]

We measured its effect on a free-threaded build and it got slightly faster, but maybe not on a default build.

there were no results for Windows though

[mdboom]

I think that this is due to some change on the benchmarking machine, not a change in the CPython source code. I re-ran the commit from 08/17 yesterday (on 8/27), and it was 7% slower than when it was first run back on (8/17). I also re-ran the commit from 08/25, and it was only 1% slower (basically within the usual amount of noise). So my best hypothesis is that something has changed on that machine in the 08/17 - 08/25 timeframe. However, I'm a bit stumped as to what the change may be.

• The Windows Update logs show the last update on 08/13 (i.e. prior to this range).

• The machine is a managed corporate machine, so it does run various security tools in the background that could be interfering, and they may have been updated during that timeframe -- though Task Manager didn't show any out-of-control processes for the moment I looked.

• The compiler version has not changed in many months (we explicitly don't update it for this reason).

• It could be something thermal -- i.e. another machine was placed nearby in the rack -- unfortunately, pyperf doesn't collect thermal data on Windows like it does on Linux -- that could help confirm that hypothesis.

I have rebooted the machine, and I am running the 08/17 commit one more time to see if it remains slower or reverts back to the earlier measurement. If it is still slower, we will probably need to just consider this the new normal and recompute all of the the 3.13 and 3.14 longitudinal weeklies (which will take about 87 hours of compute, less if we're a bit selective).

The good news is that comparing results to the immediate base, which is the common use case, should still be fine and accurate (since they were measured back to back, not with a space of days or weeks in between), so it doesn't call into question any past decisions we may have made about PRs.

---

Separately, to close the thread on @itamaro's hypothesis:
I measured python/cpython#122975, and it seems to have no change on a non-free-threaded build. I couldn't measure it on a free-threaded build on Windows, because PGO builds there are broken, known issue.

[mdboom]

After rebooting the machine, I am getting results for the 08/17 commit that closely match what was measured originally, so maybe that's all it needed. At this point my plan is to wait until the next weekly on Sunday. Assuming that puts Windows back in the ballpark of no slowness, I will regenerate only the Windows measurements between 8/17 and today, 8/28.

[mdboom]

It's clear that rebooting the machine resolved the issue. Marking this as closed, and I will delete the bad data points between 8-17 and 8-28.