add azure CI, temporarily disable others #107

mattip · 2020-11-29T21:59:03Z

Add azure matrix based on .travis.yml and azure-posix.yml based on the one in numpy-wheels
disable other CI services
still need to create new tokens in the Azure Pipelines GUI

tylerjereddy · 2020-11-30T03:06:09Z

Thanks for getting this started Matti

mattip · 2020-11-30T08:47:27Z

By mistake I tried with Cython-3.0a6 which failed. That is worrisome

Edit: add link

Cython fail

Processing scipy/optimize/_group_columns.pyx
Traceback (most recent call last):
  File "/opt/python/cp37-cp37m/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 711, in setuptools_main
    return main(command_line = 1)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 729, in main
    result = compile(sources, options)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 628, in compile
    return compile_multiple(source, options)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 605, in compile_multiple
    result = run_pipeline(source, options, context=context)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Main.py", line 494, in run_pipeline
    pipeline = Pipeline.create_pyx_pipeline(context, options, result)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 252, in create_pyx_pipeline
    create_pipeline(context, mode, exclude_classes=exclude_classes),
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Pipeline.py", line 144, in create_pipeline
    from .ParseTreeTransforms import WithTransform, NormalizeTree, PostParse, PxdPostParse
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1632, in <module>
    class AnalyseDeclarationsTransform(EnvTransform):
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 1640, in AnalyseDeclarationsTransform
    """, level='c_class', pipeline=[NormalizeTree(None)])
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/TreeFragment.py", line 235, in __init__
    mod = t = parse_from_strings(name, fmt_code, fmt_pxds, level=level, initial_pos=initial_pos)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/TreeFragment.py", line 75, in parse_from_strings
    scope = scope, context = context, initial_pos = initial_pos)
  File "Cython/Compiler/Scanning.py", line 306, in Cython.Compiler.Scanning.PyrexScanner.__init__
  File "Cython/Compiler/Scanning.py", line 35, in Cython.Compiler.Scanning.get_lexicon
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Compiler/Lexicon.py", line 147, in make_lexicon
    (Eof, Method('eof_action'))
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Plex/Lexicons.py", line 129, in __init__
    nfa, default_initial_state, spec, token_number)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Plex/Lexicons.py", line 162, in add_token_to_machine
    match_bol=1, nocase=0)
  File "/opt/python/cp37-cp37m/lib/python3.7/site-packages/Cython/Plex/Regexps.py", line 301, in build_machine
  File "Cython/Plex/Transitions.py", line 60, in Cython.Plex.Transitions.TransitionMap.add
  File "Cython/Plex/Machines.py", line 124, in Cython.Plex.Machines.Node.__hash__
OverflowError: Python int too large to convert to C ssize_t

mattip · 2020-11-30T13:52:25Z

The 9 azure builds of (linux64, linux32, macos) * (python3.7, 3.8, 3.9) are green. I have set the repo up to build weekly at 3:27am UTC every Sunday. as well as for PRs.

@tylerjereddy please take a look before I add tokens for uploading.

tylerjereddy

Looks pretty solid to me--I added one question inline.

I did three spot checks for new CI test suite results vs. the same jobs on Travis CI, and the skipped/xfailed/xpassed results look quite close, which is a good sign for i.e., matching deps/setups:

64-bit Linux Python 3.9
    Old Travis
        40928 passed, 2252 skipped, 101 xfailed, 8 xpassed, 40 warnings in 1198.41s (0:19:58) =
    Azure
        41046 passed, 2260 skipped, 101 xfailed, 8 xpassed, 40 warnings in 627.73s (0:10:27) =
32-bit linux Python 3.7
    Old Travis
        40911 passed, 2260 skipped, 113 xfailed, 5 xpassed, 28 warnings in 992.99s (0:16:32) 
    Azure
        41039 passed, 2258 skipped, 114 xfailed, 4 xpassed, 40 warnings in 759.56s (0:12:39) = 
MacOS Python 3.8
    Old Travis
         30150 passed, 2067 skipped, 75 xfailed, 6 xpassed, 28 warnings in 589.91s (0:09:49) =
    Azure
         30274 passed, 2068 skipped, 75 xfailed, 6 xpassed, 28 warnings in 240.53s (0:04:00) =

The total numbers of passing tests are less relevant because that'll vary with the master branch commit more easily, but even those are close looking to me.

Not sure if @rgommers has any additional comments, but seems like we can move forward with the token setup now?

tylerjereddy · 2020-12-02T02:11:38Z

azure-posix.yml

+
+      - task: UsePythonVersion@0
+        inputs:
+          versionSpec: $(AZURE_PYTHON_VERSION)


When is this version of the UsePythonVersion task used? It looks like we always set the one for MB_PYTHON_VERSION below in any case?

I guess AZURE_PYTHON_VERSION is something I don't understand yet--is it built-in to Azure or coming from multibuild?

It is not from multibuild. Let me try to delete those steps. It came from MacPython/numpy-wheels#99, I think the comment "There is a problem with the Azure linux builds: for some reason it is setting the python version in the host machine even though that is not needed. I have a fix for that." was relevant but I cannot recall what it means.

Maybe it was from a time where the default python is 2.7, so we need to set something, but could not set 3.9 because Azure did not have it yet?

It is needed for macOSx

rgommers · 2020-12-02T12:09:06Z

Looks great to me, thanks for working on this @mattip.

mattip · 2020-12-02T23:01:31Z

The upload step is failing but I think it is because this is a branch in my repo. So how to proceed without burning too many credits? Put back appveyor and merge this with travis disabled, and then enable travis in another PR after we see the uploads succeed?

tylerjereddy

Put back appveyor and merge this with travis disabled, and then enable travis in another PR after we see the uploads succeed?

+1 from me on that idea; also, there may be one line accidentally deleted in the Travis file? I commented inline for you to double check.

Apart from that, I think you can self-merge with Appveyor reactivated.

tylerjereddy · 2020-12-03T03:14:27Z

_travis.yml.backup

@@ -6,7 +6,6 @@ env:
        - PLAT=x86_64
        - NP_BUILD_DEP="numpy==1.16.5"
        - CYTHON_BUILD_DEP="Cython==0.29.18"
-        - PYBIND11_BUILD_DEP="pybind11==2.4.3"


I think this is still used by a line that hasn't been deleted?

thanks. I reverted this change and changed the NP_* requirement use oldest-supported-numpy instead, like on azure.

mattip · 2020-12-03T16:38:30Z

"only" the upload is failing, so I will merge this with the travis runs disabled.

mattip added 2 commits November 30, 2020 00:12

add azure CI, temporarily disable others

4702432

remove cruft from numpy-wheels

fb450b9

mattip added 2 commits November 30, 2020 08:45

add TEST_DEPENDS

9b2dd12

depend on pyproject.toml for build, move TEST_DEPENDS

3fca188

mattip added 4 commits November 30, 2020 10:47

use BUILD_DEPENDS copied from pyproject.toml

36e7cde

upgrade setuptools, add more verbose scripting

6e1c6b0

use PYTHON_EXE not python

b9213e7

define PIP_CMD

47b811f

tylerjereddy reviewed Dec 2, 2020

View reviewed changes

add tokens, remove linux. macos from travis

4adda6f

tylerjereddy reviewed Dec 3, 2020

View reviewed changes

tweak travis.yml backup, revive appveyor.yml

61155fe

mattip merged commit 223b156 into MacPython:master Dec 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add azure CI, temporarily disable others #107

add azure CI, temporarily disable others #107

Uh oh!

mattip commented Nov 29, 2020

Uh oh!

tylerjereddy commented Nov 30, 2020

Uh oh!

mattip commented Nov 30, 2020 •

edited

Loading

Uh oh!

mattip commented Nov 30, 2020

Uh oh!

tylerjereddy left a comment

Uh oh!

tylerjereddy Dec 2, 2020

Uh oh!

mattip Dec 2, 2020

Uh oh!

mattip Dec 2, 2020

Uh oh!

mattip Dec 2, 2020

Uh oh!

rgommers commented Dec 2, 2020

Uh oh!

mattip commented Dec 2, 2020

Uh oh!

tylerjereddy left a comment

Uh oh!

tylerjereddy Dec 3, 2020

Uh oh!

mattip Dec 3, 2020

Uh oh!

mattip commented Dec 3, 2020

Uh oh!

Uh oh!

add azure CI, temporarily disable others #107

add azure CI, temporarily disable others #107

Uh oh!

Conversation

mattip commented Nov 29, 2020

Uh oh!

tylerjereddy commented Nov 30, 2020

Uh oh!

mattip commented Nov 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Nov 30, 2020

Uh oh!

tylerjereddy left a comment

Choose a reason for hiding this comment

Uh oh!

tylerjereddy Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

mattip Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

mattip Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

mattip Dec 2, 2020

Choose a reason for hiding this comment

Uh oh!

rgommers commented Dec 2, 2020

Uh oh!

mattip commented Dec 2, 2020

Uh oh!

tylerjereddy left a comment

Choose a reason for hiding this comment

Uh oh!

tylerjereddy Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

mattip Dec 3, 2020

Choose a reason for hiding this comment

Uh oh!

mattip commented Dec 3, 2020

Uh oh!

Uh oh!

mattip commented Nov 30, 2020 •

edited

Loading