Consider replacing AppVeyor with Azure Pipelines

[JukkaL]

We seem to have too little capacity on AppVeyor as CI builds are often queued for a long time. Azure Pipelines apparently supports Windows and 10 parallel jobs, for free. It might be a better alternative for mypy, since our builds are somewhat slow.

Is anybody interested in looking at this? Any help would be appreciated, even just small experiments and helpful pointers.

Here's a blog post that talks about the open source offering: https://azure.microsoft.com/en-gb/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

[JukkaL]

This is very important for overall productivity, and having more capacity could allow us to set up additional Windows builds. I'm not quite happy with our current state of testing on Windows.

[emmatyping]

I know @asottile has experience with Azure pipelines, perhaps he has some experience he can share.

I personally have been experimenting with Github actions, which allows up to 20 concurrent workers (https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-github-actions#usage-limits).

[asottile]

at this point, the offerings from azure pipelines and github actions are essentially equivalent -- they use the same backing builder machinery as far as I can tell as well. the only difference is the number of parallel builders (GA: 20, AP: 10)

I've set up a quick demo of what this would look like if you took your appveyor.yml and naively ported it to a github action: https://github.com/asottile/mypy/pull/1

let me know if you'd like me to PR this against the main repo and I can clean it up a little bit (s/test/main/ mainly + git rm appveyor.yml + whatever else you'd like me to do + the EXTRA_ARGS stuff that I forgot about)

[asottile]

and yes, successful nerd snipe ethan 🙃

[emmatyping]

Oh that is great, thank you! And uh, sorry for the nerd snipe :P

It seems that there is an issue in the link phase of building the mypyc tests, I've reached out to Github support about that. Seems like it should "just work".

[JukkaL]

@asottile Wonderful, thank you! A good next step once the build issue has been sorted out would be to run it along with Appveyor in the main repo for a while to evaluate relative performance and reliability. If everything looks good, we can drop Appveyor after a while.

[asottile]

when I set passenv = * in tox.ini the failures drop from 235 to 1 -- so there's some environment variable that's necessary to make those work

I can try and debug this further later -- will probably use this to narrow down what's necessary -- probably with testIncrementalCompilation6_separate (chosen haphazardly as one of the ones which fails without that and succeeds with it)

[asottile]

this appears to be the minimal set of environment variables necessary:

['PATH', 'PROGRAMDATA', 'PROGRAMFILES(X86)', 'SYSTEMROOT']

PATH and SYSTEMROOT are sent by default from tox -- so PROGRAMDATA and PROGRAMFILES(X86) will need to be added to passenv

[asottile]

2020-01-17T21:42:49.9175964Z ================================== FAILURES ===================================
2020-01-17T21:42:49.9176855Z ________________________________ testDictFree _________________________________
2020-01-17T21:42:49.9177779Z [gw1] win32 -- Python 3.7.6 D:\a\mypy\mypy\.tox\py37\Scripts\python.EXE
2020-01-17T21:42:49.9178633Z data: D:\a\mypy\mypy\mypyc\test-data\run.test:4408:
2020-01-17T21:42:49.9179149Z D:\a\mypy\mypy\mypyc\test\test_run.py:115: in run_case
2020-01-17T21:42:49.9179442Z     self.run_case_inner(testcase)
2020-01-17T21:42:49.9179699Z D:\a\mypy\mypy\mypyc\test\test_run.py:130: in run_case_inner
2020-01-17T21:42:49.9179960Z     self.run_case_step(testcase, step)
2020-01-17T21:42:49.9180936Z D:\a\mypy\mypy\mypyc\test\test_run.py:286: in run_case_step
2020-01-17T21:42:49.9181261Z     assert_test_output(testcase, outlines, msg, expected)
2020-01-17T21:42:49.9181562Z D:\a\mypy\mypy\mypyc\test\testutil.py:155: in assert_test_output
2020-01-17T21:42:49.9181851Z     '{} ({}, line {})'.format(message, testcase.file, testcase.line))
2020-01-17T21:42:49.9182172Z D:\a\mypy\mypy\mypy\test\helpers.py:116: in assert_string_arrays_equal
2020-01-17T21:42:49.9182450Z     raise AssertionError(msg)
2020-01-17T21:42:49.9182822Z E   AssertionError: Invalid output (D:\a\mypy\mypy\mypyc\test-data\run.test, line 4408)
2020-01-17T21:42:49.9183129Z ---------------------------- Captured stdout call -----------------------------
2020-01-17T21:42:49.9183404Z running build_ext
2020-01-17T21:42:49.9183647Z building 'native' extension
2020-01-17T21:42:49.9183898Z creating build\temp.win-amd64-3.7
2020-01-17T21:42:49.9184099Z creating build\temp.win-amd64-3.7\Release
2020-01-17T21:42:49.9184362Z creating build\temp.win-amd64-3.7\Release\build
2020-01-17T21:42:49.9184901Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -ID:\a\mypy\mypy\mypyc\lib-rt -Ic:\hostedtoolcache\windows\python\3.7.6\x64\include -Ic:\hostedtoolcache\windows\python\3.7.6\x64\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcbuild\__native.c /Fobuild\temp.win-amd64-3.7\Release\build\__native.obj /O2 /wd4102 /wd4101 /wd4146
2020-01-17T21:42:49.9185340Z __native.c
2020-01-17T21:42:49.9185493Z 
2020-01-17T21:42:49.9186199Z D:\a\mypy\mypy\mypyc\lib-rt\getargs.c(1432): warning C4244: '=': conversion from 'Py_ssize_t' to 'int', possible loss of data
2020-01-17T21:42:49.9186453Z 
2020-01-17T21:42:49.9186718Z creating C:\Users\RUNNER~1\AppData\Local\Temp\mypy-test-rmu38wl8\tmp\build\lib.win-amd64-3.7
2020-01-17T21:42:49.9187393Z C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\hostedtoolcache\windows\python\3.7.6\x64\Libs /LIBPATH:D:\a\mypy\mypy\.tox\py37\libs /LIBPATH:D:\a\mypy\mypy\.tox\py37\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.24.28314\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_native build\temp.win-amd64-3.7\Release\build\__native.obj /OUT:build\lib.win-amd64-3.7\native.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\build\native.cp37-win_amd64.lib
2020-01-17T21:42:49.9187803Z    Creating library build\temp.win-amd64-3.7\Release\build\native.cp37-win_amd64.lib and object build\temp.win-amd64-3.7\Release\build\native.cp37-win_amd64.exp
2020-01-17T21:42:49.9188064Z 
2020-01-17T21:42:49.9188321Z Generating code
2020-01-17T21:42:49.9188479Z 
2020-01-17T21:42:49.9188729Z Finished generating code
2020-01-17T21:42:49.9188889Z 
2020-01-17T21:42:49.9189182Z copying build\lib.win-amd64-3.7\native.cp37-win_amd64.pyd -> 
2020-01-17T21:42:49.9189344Z 
2020-01-17T21:42:49.9189592Z *** Exit status: 1
2020-01-17T21:42:49.9190441Z ---------------------------- Captured stderr call -----------------------------
2020-01-17T21:42:49.9190761Z Expected:
2020-01-17T21:42:49.9191037Z Actual:
2020-01-17T21:42:49.9191306Z   Traceback (most recent call last):            (diff)
2020-01-17T21:42:49.9195768Z     File "driver.py", line 12, in <module>      (diff)
2020-01-17T21:42:49.9196613Z       test()                                    (diff)
2020-01-17T21:42:49.9197328Z     File "driver.py", line 10, in test          (diff)
2020-01-17T21:42:49.9197630Z       assert x == y                             (diff)
2020-01-17T21:42:49.9197839Z   AssertionError                                (diff)
2020-01-17T21:42:49.9198123Z 

any hints on this one? it's the only remaining failing test

my guess is this is an actual failure -- looks like it's crashing the compiler on this newer version of visual studio?

[emmatyping]

@asottile hmm that is odd, perhaps @msullivan would be better at diagnosing than me. But it seems that setuptools gets through building the library since it gets to copying build\lib.win-amd64-3.7\native.cp37-win_amd64.pyd -> . What makes you think it is crashing the compiler?

It seems to me it is failing the GC test on this line. I have no idea why.

[JukkaL]

We don't need to block enabling github actions on the test failure. It would be okay to skip the test temporarily, as long as we re-enable it before the next release.

[msullivan]

That is weird but we can disable it for now. It is a very fragile seeming test so to be honest it breaking is not super surprising

[asottile]

awesome, I'll make a PR when I'm at a computer

[msullivan]

I did some experimenting and it looks like somehow something is getting freed between the first and second get_objects calls. This is all in interpreted code and happens even if we don't import or call the compiled code, so I'm just going to remove the test which is too fragile. Probably anything involving gc.get_objects() is going to be more fragile than it is worth.