Skip to content

Combine test workflows into one #5623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 20, 2022

Conversation

michaelosthege
Copy link
Member

@michaelosthege michaelosthege commented Mar 19, 2022

This PR combines pytest, arviz-compat and jaxtests workflows into one (big) tests workflow.

Motivations

Changes

  • ✔ Removed the buildosx build script. Chris confirmed via Slack that it can be removed.
  • arviz-compat and jaxtests were deleted - their jobs were moved into tests (previously pytest.
  • ✔ Upgraded the tests from ubuntu 18.04 to 20.04 which is the most recent LTS release.
  • ✔ Redistributed tests between jobs to reduce the runtime (closes Combine windows workflows #5475)

The steps within each job were not changed in this commit.
@codecov
Copy link

codecov bot commented Mar 19, 2022

Codecov Report

Merging #5623 (c3c383f) into main (849a64d) will decrease coverage by 0.03%.
The diff coverage is n/a.

❗ Current head c3c383f differs from pull request most recent head a9955ac. Consider uploading reports for the commit a9955ac to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #5623      +/-   ##
==========================================
- Coverage   87.67%   87.63%   -0.04%     
==========================================
  Files          76       76              
  Lines       13717    13717              
==========================================
- Hits        12026    12021       -5     
- Misses       1691     1696       +5     
Impacted Files Coverage Δ
pymc/smc/smc.py 96.38% <0.00%> (-1.61%) ⬇️
pymc/util.py 76.25% <0.00%> (-1.25%) ⬇️
pymc/sampling.py 85.97% <0.00%> (-0.12%) ⬇️
pymc/distributions/simulator.py 87.50% <0.00%> (+1.38%) ⬆️

@michaelosthege michaelosthege marked this pull request as ready for review March 19, 2022 21:13
@ricardoV94
Copy link
Member

Looking at the runtime there are 4 or 5 workflows that took ~6 minutes to run, one of which only spent 45 seconds running actual tests, and the rest was setting up the environment

https://github.com/pymc-devs/pymc/runs/5612971573?check_suite_focus=true

I would merge those short workflows with some of the others that take like 11 minutes. We still have some workflows that take ~30 minutes, so we will not loose anything by merging the fast ones.

@michaelosthege michaelosthege force-pushed the ci-refactor branch 2 times, most recently from c44de15 to b320df9 Compare March 20, 2022 00:51
@michaelosthege
Copy link
Member Author

michaelosthege commented Mar 20, 2022

I aimed for 15-20 minutes per job. Only the test_distribution.py still takes much longer. But I don't want to split it up in this PR.

I was about to comment final success, but then I noticed that the Windows jobs were only running the first item in each subset 😮 And that was already the case on main 😲
I home my latest push fixes it, but in any case I'll probably have to do more balancing.

Some stats compared to 849a64d:

  • Number of test jobs reduced from 23 to 22
  • Total runtime reduced from 6h38m in to 6h22m

@michaelosthege michaelosthege force-pushed the ci-refactor branch 3 times, most recently from 6a8b2f7 to c3c383f Compare March 20, 2022 09:58
Copy link
Member

@ricardoV94 ricardoV94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,

This also explains why the windows xfails were not failing with strict XPASS in the past

@ricardoV94
Copy link
Member

ricardoV94 commented Mar 20, 2022

Failing test seems to be #5575

I'll try to fix it this week. Shouldn't be a blocker here

@michaelosthege
Copy link
Member Author

I noticed that the test_distribution_moments.py takes 12 minutes, which is much longer than I would have expected.

My last force-push moved mixture tests between the ubuntu groups to shave the longest job runtime down.

I'm going to rebase-merge since there's no need to run the CI again.

@michaelosthege michaelosthege merged commit b895e40 into pymc-devs:main Mar 20, 2022
@michaelosthege michaelosthege deleted the ci-refactor branch March 20, 2022 11:31
@ricardoV94 ricardoV94 mentioned this pull request Mar 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Combine windows workflows
2 participants