From c849db59ab40b82d1023990e45f822063f9a974a Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Mon, 14 Mar 2022 12:18:33 +0000 Subject: [PATCH 1/8] maint: update CI wheel building script Previously it wasn't possible to build python_flint with the most recent versions of Cython but that was fixed in b3eb3547. That makes it possible to update all build dependencies and to create wheels for newer Python versions. This commit updates the build scripts that build wheels for OSX and Linux in GitHub Actions. The result of these changes is that the latest versions of all build and runtime dependencies are currently being used and x86_64 bit wheels are being built for OSX and Linux for all supported CPython versions. - Unpin the version of Cython used to build python_flint so that the newest version will always be used. At the time of writing that means using Cython 0.29.28 but newer versions will automatically be used when released. - Build 64 bit CPython wheels for OSX and Linux for all currently supported Python versions (3.7, 3.8, 3.9 and 3.10). - Fix the YASM version from 1.3.1 to 1.3.0. There does not seem to be a 1.3.1 version of YASM any more and 1.3.0 appears to be the newest version. It is unclear what happened to 1.3.1 which was previously used successfully. - Use the cibuildwheel GitHub Action for wheel building and bump version of cibuildwheel to 2.3.1 to get support for Python 3.10. - Bump versions of flint and arb to the latest releases (flint 2.8.4 and arb 2.22.0). - Check that all other dependencies are at their latest released versions (GMP 6.2.1 and MPIR 3.0.0 and MPFR 4.1.0). - Wheels are still not built for Windows - this is the biggest omission. There are conda-forge recipes for building python-flint and it should be possible to adapt those. - Newer OSX runs on Apple's M1 hardware so the OSX wheels built here will not work there. It is not yet possible to run M1 hardware in GitHub Actions so it is not immediately clear how to build those wheels. - It is also possible to build 32 bit Linux wheels but these are less likely to be used. Building them significantly increases the build time because numpy does not provide 32 bit wheels and numpy is currently a build-time dependency. Linux wheels also consume a lot of disk space which is limited for artifacts in GitHub Actions. - I have not checked whether it is possible to build wheels for pypy which is another likely platform that could be supported. - The build-time dependency on numpy can probably be eliminated. A couple of small things are being used from numpy.distutils which is itself probably going to be deprecated in future. --- .github/workflows/buildwheel.yml | 12 ++++++------ bin/build_variables.sh | 6 +++--- bin/build_wheel.sh | 4 ++-- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.github/workflows/buildwheel.yml b/.github/workflows/buildwheel.yml index d8cdb150..b94675b3 100644 --- a/.github/workflows/buildwheel.yml +++ b/.github/workflows/buildwheel.yml @@ -15,20 +15,20 @@ jobs: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 - - name: Install cibuildwheel - run: python -m pip install cibuildwheel==1.10.0 - name: Build wheels - run: python -m cibuildwheel --output-dir wheelhouse + uses: pypa/cibuildwheel@v2.3.1 env: - CIBW_BUILD: cp37-* cp38-* + CIBW_BUILD: cp37-* cp38-* cp39-* cp310-* + CIBW_SKIP: "*-win32 *-manylinux_i686 *-musllinux_*" CIBW_BEFORE_ALL_LINUX: bin/cibw_before_build_linux.sh CIBW_BEFORE_ALL_MACOS: bin/cibw_before_build_macosx.sh - # There are problems with both older and newer cython versions... - CIBW_BEFORE_BUILD: pip install numpy cython==0.27.3 + CIBW_BEFORE_BUILD: pip install numpy cython CIBW_ENVIRONMENT: > C_INCLUDE_PATH=$(pwd)/.local/include/ LIBRARY_PATH=$(pwd)/.local/lib/ LD_LIBRARY_PATH=$(pwd)/.local/lib:$LD_LIBRARY_PATH + CIBW_TEST_COMMAND: "python -c \'import flint; assert str(flint.fmpq(3, 2)**2) == \"9/4\"\'" + - uses: actions/upload-artifact@v2 with: path: wheelhouse/*.whl diff --git a/bin/build_variables.sh b/bin/build_variables.sh index 027532c4..253f787e 100644 --- a/bin/build_variables.sh +++ b/bin/build_variables.sh @@ -14,8 +14,8 @@ PREFIX=$(pwd)/.local mkdir -p $PREFIX GMPVER=6.2.1 -YASMVER=1.3.1 +YASMVER=1.3.0 MPIRVER=3.0.0 MPFRVER=4.1.0 -FLINTVER=2.7.1 -ARBVER=2.19.0 +FLINTVER=2.8.4 +ARBVER=2.22.0 diff --git a/bin/build_wheel.sh b/bin/build_wheel.sh index ba314f40..bc4c62ef 100755 --- a/bin/build_wheel.sh +++ b/bin/build_wheel.sh @@ -12,8 +12,8 @@ source bin/build_variables.sh python3 -m venv $PREFIX/venv source $PREFIX/venv/bin/activate pip install -U pip wheel delocate -pip install numpy cython==0.27.3 -# N.B. bugs in both older and newer Cython versions... +pip install numpy cython +# Working as of cython==0.29.28 C_INCLUDE_PATH=.local/include/ LIBRARY_PATH=.local/lib/ pip wheel . From ae6a900629f04168b7b1150f3221d1996ddf4b88 Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Thu, 17 Mar 2022 17:03:48 +0000 Subject: [PATCH 2/8] maint: add jobs to test wheels in CI Add jobs to the wheel build GitHub Actions workflow that will install and test each built wheel in a fresh test runner. For each combination of Python version and OS (currently 3.7-3.10 and OSX/Linux) this will run a job that installs the wheel and runs the test script using the installed wheel. While this can give some basic confidence that the wheels do work it is still possible that these tests can pass in CI with wheels that would fail when transfered to another computer. At the time of this commit that is the case with the manylinux wheels which pass all tests but fail when installed to another Linux machine. --- .github/workflows/buildwheel.yml | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/.github/workflows/buildwheel.yml b/.github/workflows/buildwheel.yml index b94675b3..3a063683 100644 --- a/.github/workflows/buildwheel.yml +++ b/.github/workflows/buildwheel.yml @@ -7,12 +7,13 @@ jobs: name: Build wheels for ${{ matrix.os }} runs-on: ${{ matrix.os }} strategy: + fail-fast: false matrix: # os: [ubuntu-20.04, windows-2019, macOS-10.15] os: [ubuntu-20.04, macOS-10.15] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - uses: actions/setup-python@v2 - name: Build wheels @@ -27,8 +28,32 @@ jobs: C_INCLUDE_PATH=$(pwd)/.local/include/ LIBRARY_PATH=$(pwd)/.local/lib/ LD_LIBRARY_PATH=$(pwd)/.local/lib:$LD_LIBRARY_PATH - CIBW_TEST_COMMAND: "python -c \'import flint; assert str(flint.fmpq(3, 2)**2) == \"9/4\"\'" + CIBW_TEST_COMMAND: python -c 'import flint; print(str(flint.fmpz(2)))' - uses: actions/upload-artifact@v2 with: path: wheelhouse/*.whl + + test_wheels: + needs: build_wheels + name: Test ${{ matrix.python-version }} wheel on ${{ matrix.os }} + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [ubuntu-20.04, macOS-10.15] + python-version: ['3.7', '3.8', '3.9', '3.10'] + + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python-version }} + - uses: actions/download-artifact@v3 + with: + name: artifact + path: wheelhouse + - run: python -m venv venv + - run: venv/bin/pip install -U pip + - run: venv/bin/pip install --find-links wheelhouse python_flint + - run: venv/bin/python test/test.py From 98430e3f172e8243d2b0980c89a06f3f8ecfd308 Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Thu, 17 Mar 2022 17:10:56 +0000 Subject: [PATCH 3/8] maint: use MPIR for building manylinux wheels The manylinux wheels built with GMP crashed with SIGILL when used on another Linux machine even though they passed tests in CI. This commit switches to using MPIR instead of GMP for the manylinux wheels because those wheels appear to be more "portable" according to limited testing so far. Maybe there is a configure option that could be used with GMP to make it more portable... --- bin/cibw_before_build_linux.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/cibw_before_build_linux.sh b/bin/cibw_before_build_linux.sh index 01506701..cfd55dbf 100755 --- a/bin/cibw_before_build_linux.sh +++ b/bin/cibw_before_build_linux.sh @@ -2,5 +2,5 @@ yum install -y xz bin/build_dependencies_unix.sh\ - --gmp gmp\ + --gmp mpir\ --host x86_64-unknown-linux-gnu From c6d5a75aebcee49e932dc866e52bf63bc6fc319f Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Sat, 19 Mar 2022 12:12:21 +0000 Subject: [PATCH 4/8] tmp: check output of configfsf.guess --- bin/build_dependencies_unix.sh | 2 ++ bin/cibw_before_build_linux.sh | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/build_dependencies_unix.sh b/bin/build_dependencies_unix.sh index 52013c3d..ef7f08c0 100755 --- a/bin/build_dependencies_unix.sh +++ b/bin/build_dependencies_unix.sh @@ -74,6 +74,8 @@ if [ $USE_GMP = "gmp" ]; then curl -O https://gmplib.org/download/gmp/gmp-$GMPVER.tar.xz tar xf gmp-$GMPVER.tar.xz cd gmp-$GMPVER + ./configfsf.guess + exit 1 ./configure --prefix=$PREFIX\ --enable-fat\ --enable-shared=yes\ diff --git a/bin/cibw_before_build_linux.sh b/bin/cibw_before_build_linux.sh index cfd55dbf..01506701 100755 --- a/bin/cibw_before_build_linux.sh +++ b/bin/cibw_before_build_linux.sh @@ -2,5 +2,5 @@ yum install -y xz bin/build_dependencies_unix.sh\ - --gmp mpir\ + --gmp gmp\ --host x86_64-unknown-linux-gnu From 6afd07012cc85cdf30b34d4870cf92d054372a0b Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Sat, 19 Mar 2022 12:18:04 +0000 Subject: [PATCH 5/8] try fixing host for GMP --- bin/build_dependencies_unix.sh | 2 +- bin/cibw_before_build_linux.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/build_dependencies_unix.sh b/bin/build_dependencies_unix.sh index ef7f08c0..6ff7117e 100755 --- a/bin/build_dependencies_unix.sh +++ b/bin/build_dependencies_unix.sh @@ -74,8 +74,8 @@ if [ $USE_GMP = "gmp" ]; then curl -O https://gmplib.org/download/gmp/gmp-$GMPVER.tar.xz tar xf gmp-$GMPVER.tar.xz cd gmp-$GMPVER + # Show the output of configfsf.guess ./configfsf.guess - exit 1 ./configure --prefix=$PREFIX\ --enable-fat\ --enable-shared=yes\ diff --git a/bin/cibw_before_build_linux.sh b/bin/cibw_before_build_linux.sh index 01506701..e3183269 100755 --- a/bin/cibw_before_build_linux.sh +++ b/bin/cibw_before_build_linux.sh @@ -3,4 +3,4 @@ yum install -y xz bin/build_dependencies_unix.sh\ --gmp gmp\ - --host x86_64-unknown-linux-gnu + --host x86_64-pc-linux-gnu From 486a29a2ad3552028c137fdf2a589f54e41de7ce Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Sat, 19 Mar 2022 13:14:18 +0000 Subject: [PATCH 6/8] switch back to MPIR for Linux wheels --- bin/cibw_before_build_linux.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/cibw_before_build_linux.sh b/bin/cibw_before_build_linux.sh index e3183269..0bfc1456 100755 --- a/bin/cibw_before_build_linux.sh +++ b/bin/cibw_before_build_linux.sh @@ -2,5 +2,5 @@ yum install -y xz bin/build_dependencies_unix.sh\ - --gmp gmp\ + --gmp mpir\ --host x86_64-pc-linux-gnu From 1e5eb93d833c2e626ba384dfd49a4cd7c4a94998 Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Sun, 20 Mar 2022 20:00:05 +0000 Subject: [PATCH 7/8] try using manylinux2010 --- .github/workflows/buildwheel.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/buildwheel.yml b/.github/workflows/buildwheel.yml index 3a063683..1cd21771 100644 --- a/.github/workflows/buildwheel.yml +++ b/.github/workflows/buildwheel.yml @@ -21,6 +21,7 @@ jobs: env: CIBW_BUILD: cp37-* cp38-* cp39-* cp310-* CIBW_SKIP: "*-win32 *-manylinux_i686 *-musllinux_*" + CIBW_MANYLINUX_X86_64_IMAGE: manylinux2010 CIBW_BEFORE_ALL_LINUX: bin/cibw_before_build_linux.sh CIBW_BEFORE_ALL_MACOS: bin/cibw_before_build_macosx.sh CIBW_BEFORE_BUILD: pip install numpy cython From ecbaa5b0696e14a1d7f8fdd64dd10be7ee82c35e Mon Sep 17 00:00:00 2001 From: Oscar Benjamin Date: Sun, 20 Mar 2022 20:37:48 +0000 Subject: [PATCH 8/8] try manylinux2014 --- .github/workflows/buildwheel.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/buildwheel.yml b/.github/workflows/buildwheel.yml index 1cd21771..2b2da417 100644 --- a/.github/workflows/buildwheel.yml +++ b/.github/workflows/buildwheel.yml @@ -21,7 +21,7 @@ jobs: env: CIBW_BUILD: cp37-* cp38-* cp39-* cp310-* CIBW_SKIP: "*-win32 *-manylinux_i686 *-musllinux_*" - CIBW_MANYLINUX_X86_64_IMAGE: manylinux2010 + CIBW_MANYLINUX_X86_64_IMAGE: manylinux2014 CIBW_BEFORE_ALL_LINUX: bin/cibw_before_build_linux.sh CIBW_BEFORE_ALL_MACOS: bin/cibw_before_build_macosx.sh CIBW_BEFORE_BUILD: pip install numpy cython