acceptance tests

[jonathanong]

was thinking about how node created breaking changes in bugfixes very often and how to avoid it. how about a test infrastructure that tests popular packages on a nightly build? i.e. create a container with the nightly build, install express/hapi/nan/whatever and run npm test. might catch some tiny bugs before releasing.

[othiym23]

I would love this, not least for npm itself. It's only diligence on the part of @wraithan that we caught what would have completely broken npm when used with Node.js 0.11.15 had it not been caught beforehand. npm, express, leveldown, gulp, hapi, and 5-10 others would be a good place to start.

[wraithan]

A few months back I brought up this idea with @nvcexploder. Back then we were talking about hapi and the node-newrelic module. He came up with the idea that there could be a repo of tests that you just PR your module into, which if this suite existed for io.js folks could do that.

An alternative, to borrow from the rust community, would be something like http://rust-ci.org/ The idea being that you have nightly builds that you push into a PPA. Then you have a travis-ci build that pulls from that PPA and runs the tests. Finally, to make sure it gets ran on each nightly, once the PPA is updated rust-ci triggers a rebuild of the last build of each travis-ci job it knows about.

The choice to build and push to a PPA means that rust doesn't have to be built in every single job and instead gets built once and everyone tests against the same exact builds.

I'm not sure that the rust-ci solution is the best, since it means other people maintaining the suite and having to make sure that the failures are indeed because of io.js instead of because their suite sucks. But it does mean that greater coverage is more achievable. Significant drops in the number of successful builds would be blatant indicators of breaking something.

Just my 2 cents.

[mikeal]

This should be logged as a feature request on https://github.com/iojs/build

There's no contention here, I think everyone wants this, it's just a matter of prioritizing it against build's other work.