Periodic test against master and canary

[mmarchini]

I think we should have nightly/weekly test runs against master and canary. This would help us know earlier when something breaks. nodejs/node#17685 was an awesome start, but as @bnoordhuis said in the PR it won't catch semantic changes.

I'm willing to setup the infrastructure for it. I just need to know if we can have this on Travis (by using https://docs.travis-ci.com/user/cron-jobs/) or if we should introduce this on Node.js CI.

/cc @joyeecheung

[mmarchini]

We could even have a status table in the readme 😄

Version	Status
6.x
8.x
9.x
10.x
master
canary

or

Version	v6.x	v8.x	v9.x	v10.x	master	canary
Status

[joyeecheung]

There is nodejs/build#1025 by the way.

I have seen people wgetting nightlies in the scripts to run tests on nightlies with Travis, but that looks rather hacky...

[mmarchini]

I saw that PR a while ago, but it seems stalled for some time. I'll take a look and see if we can get some traction on it.

BTW, should we move our current CI infrastructure from Travis to our Jenkins?

[joyeecheung]

BTW, should we move our current CI infrastructure from Travis to our Jenkins?

@mmarchini That's the plan, but until it happens Travis is better than nothing.

[mmarchini]

I have seen people wgetting nightlies in the scripts to run tests on nightlies with Travis, but that looks rather hacky...

I looked into Travis docs and it seems like they use nvm to manage Node.js versions. Maybe we can use it to get nightlies in a not-so-hacky way. I think I'll try to add something to Travis while we don't have support for llnode on Jenkins.

[mmarchini]

Proof of Concept: https://github.com/mmarchini/llnode

I've also removed Node.js v4.x, added v10.x (which is broken 😢) and added Mac OS X builds for v6.x, v8.x, v9.x, and v10.x. .travis.yaml grew a lot, but I think it's worth. Nightlies are ignores in the final CI result.

Setting up Cron jobs is straightforward, but need admin access to Travis I think. It can be done in the menu Settings at the end of the page.

I'll open a PR tomorrow. First I wanted to test the Cron thingy.

P.S.: I had to use https://github.com/bjfish/travis-matrix-badges to get badges for each Job since Travis don't have this feature.

[joyeecheung]

Setting up Cron jobs is straightforward, but need admin access to Travis I think. It can be done in the menu Settings at the end of the page.

Last time I checked we cannot access the admin page of Travis (mainly because we do not authorize Travis or any other third-party services to read the nodejs organization - that's also the reason why we have to use the GitHub bot to update CI status). Now this restriction makes less sense since we've moved the security repos out of this organization though.

[mmarchini]

Last time I checked we cannot access the admin page of Travis

Uh, ok, nightly tests will have to wait for Jenkins then. But we could still update our travis.yaml to cover more versions while Jenkins is not ready for llnode. I'll open a PR later today.

[mmarchini]

Crazyish idea: maaaaybe we could contact Travis support, explain the situation and see if they could enable daily Cron jobs against master for us 😶

[mmarchini]

@joyeecheung now that we can properly enable Travis in org repositories, should we still migrate to Jenkins or should we stick with Travis?

I think it's a good idea to stay with Travis since it will not inflict more burden on the Build WG and also make our setup a lot easier.

[joyeecheung]

@mmarchini We can keep Travis because it's automatic, but enabling Jenkins would allow us to test it on more platforms.

(Also I think we are trying to add Travis to the core as well, just for a faster check)