Optimize CI runs, by avoiding unnecessary rebuilds #2833
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
misl6
changed the title
misl6
added a commit
that referenced
this pull request
Sep 17, 2023
* Update `cffi` recipe for Python 3.10 (#2800) * Update __init__.py version bump to 1.15.1 * Update disable-pkg-config.patch adjust patch for 1.15.1 * Use build rather than pep517 for building (#2784) pep517 has been renamed to pyproject-hooks, and as a consequence all of the deprecated functionality has been removed. build now provides the functionality required, and since we are only interested in the metadata, we can leverage a helper function for that. I've also removed all of the subprocess machinery for calling the wrapping function, since it appears to not be as noisy as pep517. * Bump actions/setup-python and actions/checkout versions, as old ones are deprecated (#2827) * Removes `mysqldb` recipe as does not support Python 3 (#2828) * Removes `Babel` recipe as it's not needed anymore. (#2826) * Remove dateutil recipe, as it's not needed anymore (#2829) * Optimize CI runs, by avoiding unnecessary rebuilds (#2833) * Remove `pytz` recipe, as it's not needed anymore (#2830) * `freetype` recipe: Changed the url to use https as http doesn't work (#2846) * Fix `vlc` recipe build (#2841) * Correct sys_platform (#2852) On Window, sys.platform = "win32". I think "nt" is a reference to os.name. * Fix code string - quickstart.rst * Bump `kivy` version to `2.2.1` (#2855) * Use a pinned version of `Cython` for now, as most of the recipes are incompatible with `Cython==3.x.x` (#2862) * Automatically generate required pre-requisites (#2858) `get_required_prerequisites()` maintains a list of Prerequisites required by each platform. But that same information is already stored in each Prerequisite class. Rather than rather than maintaining two lists which might become inconsistent, auto-generate one. * Use `platform.uname` instead of `os.uname` (#2857) Advantages: - Works cross platform, not just Unix. - Is a namedtuple, so can use meaningful fieldnames. Also snuck in correction to typo in Readme which doesn't warrant its own review. * Fix simple typos in comments (#2863) One typo I introduced while trying to fix the other. * `build_platform` should be all-lowercase (#2864) * Docs: Fix typos and improved README quickstart (#2860) * Made p4a apk build command more general - readme * Fix twice spelled - readme * updated p4a build command * Cleanup `patching.py` (#2868) Major changes to comments, param names, function ordering. Removed deprecated LooseVersion Move os.uname to platform.uname Added win32 check Windows fix * Update Python versions in CI tests Python 3.11 was released October 2022. Python 3.7 went end-of-life June 2023 * Update documentation to reflect Python 3.7 being end-of-life. * Correct check for `--sdk` option (#2870) * Factor out dependency checking. Use modern version handling (#2866) LooseVersion used again Handle bad SDK versions * Linter fixes (#2874) The errors were: ``` pythonforandroid/bootstrap.py:136:5: F811 redefinition of unused 'name' from line 73 pythonforandroid/build.py:111:5: F811 redefinition of unused 'libs_dir' from line 82 pythonforandroid/build.py:127:5: F811 redefinition of unused 'aars_dir' from line 83 pythonforandroid/graph.py:48:12: E721 do not compare types, for exact checks use `is` pythonforandroid/graph.py:163:20: E721 do not compare types, for exact checks use `is` tests/test_build.py:39:41: E231 missing whitespace after ',' tests/test_build.py:40:58: E231 missing whitespace after ',' tests/test_build.py:41:61: E231 missing whitespace after ',' tests/test_build.py:42:71: E231 missing whitespace after ',' ``` * Remove deprecated FlatDir in Gradle template (#2876) Based on lessons from https://stackoverflow.com/questions/68215302/using-flatdirs-should-be-avoided-because-it-doesnt-support-any-meta-data-format * Standardise `ensure_dir` and `rmdir` (#2871) * Standardise ensure_dir and rmdir * Standardise ensure_dir and rmdir * Add libmysqlclient to broken list * Libtorrent failing to be rebuilt * Add boost to broken recipes list * Standardise on move (files, directories) (#2884) * Add new util function (and tests) called `move`. * Change references to sh.mv to use move (as it is faster and cross-platform). * Change conditional "mv -t" to a for loop. * Standardise on touch files (#2886) Existing code shells out to run the `touch` Unix command. This can be done faster (and cross-platform) with `pathlib`'s touch method. * Created a utility to accept string paths or pathlib paths and applies touch. * Created test for it. * Update all instances to use it, and updated their tests. * Update CHANGELOG.md and bump version to 2023.09.16 --------- Co-authored-by: HyTurtle <[email protected]> Co-authored-by: Steve Kowalik <[email protected]> Co-authored-by: Mathias Lindström <[email protected]> Co-authored-by: Ansh Dadwal <[email protected]> Co-authored-by: Julian <[email protected]> Co-authored-by: Kulothungan U.G <[email protected]> Co-authored-by: Andre Miras <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Well, parallelizing builds, even if most of the work is duplicated, is usually good. (If you're not paying for CI runs)
This is something valid also for
python-for-androidartifacts builds, as parallelizingAPKandAABbuilds takes ~14 minutes less than a build that performs APK (debug) + AAB (release) build in the same run.But ... even if we're not paying for CI runs (thank you GitHub), workers are capped at a maximum concurrent number (and in the case of our Apple Silicon M1 self-hosted runner, only one instance per time is available), which means that other jobs need to wait the completion of previous ones. (making it slower than non-parallelized builds)
Some data, on Ubuntu:
1h 19m 47s(keeps only 1 instance locked for the entire build)1h 5m 21s+ AAB1h 8m 26s(keeps 2 instances locked for the entire build)This PR merges APK and AAB builds, avoiding unnecessary rebuilds.
A lot more could be done to improve our CI wait time, but that's a start.