Spurious failure with stalled macOS builds (specifically the `check i686-apple-darwin` job)

[kennytm]

First seen on 2017 Aug 31st in:

• Fix end region emission order #44129 (comment) (check i686)
• Use hir::ItemLocalId instead of ast::NodeId in rustc::middle::region::CodeExtent. #44171 (comment) (check i686)
• Fix end region emission order #44129 (comment) (check i686 again)

Symptom:

The job just stopped in the middle with no output in 30 minutes without any reason. End of the log looks like:

[00:40:25] �[m�[m�[32m�[1m   Compiling�[m rustc_llvm v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_llvm)
[00:40:31] �[m�[m�[32m�[1m   Compiling�[m flate2 v0.2.19
[00:40:34] �[m�[m�[32m�[1m   Compiling�[m rustc_errors v0.0.0 (file:///Users/travis/build/rust-lang/rust/src/librustc_errors)



No output has been received in the last 30m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

Status of Travis and MacStadium are both green on 2017 Aug 31st. Not sure if unreported upstream issue or something else.

[ssbarnea]

Well, I was hit by the same issue using brew upgrade .... failed on rust 1.20 ... seems like a recurring problem with rust.

[alexcrichton]

I talked with Travis yesterday and got:

both your mac pros are healthy, once has two VMs and the other has three
both our networking and SAN stuff is hitting some limits, so it may be one of those, unfortunately.

which makes me think it's probably upstream issues and we don't really have a way to work around :(

[kennytm]

Not sure if Travis CI is aware of this, but the problem is very likely in the connection to the log collector or RabbitMQ, not the VMs themselves (yeah not something we can fix).

I'm currently monitoring https://travis-ci.org/rust-lang/rust/jobs/272020017, and logs are being written out if you "Follow log" at the web interface, but opening it again the logs will be all gone.

---

🤔 The failing jobs are always the check jobs, not the dist jobs. The difference between them are xcode8.2 vs xcode7.3. I wonder if this can be fixed by upgrading the check jobs to xcode8.3 or xcode9. (I don't think we want to downgrade to xcode7.3.)