cargo panics with "timeout in poll?" error from jobserver #4643

durka · 2017-10-19T03:21:43Z

Here's a log trying to build clippy: https://gist.github.com/b721066be81601774e16087210ad1919

The error happens every time at a random point while building dependencies. The only way to fix it is to disable parallelism with -j 1.

This might be a bug in jobserver, or OSX itself. I'm on OSX 10.9.5.

The text was updated successfully, but these errors were encountered:

alexcrichton · 2017-10-19T14:37:26Z

I can't find anything on google that points to 10.9 specifically (everyone seems to claim 10.12 is broken) and all the "brokenness" appears to be in different ways? If we want to fix this we probably just want to have a fallback to select in cases like this.

durka · 2017-10-19T16:18:15Z

Yeah, all I found was the 10.12 stuff as well. I couldn't get a simple repro by just calling libc::poll. Do you have any idea what kind of scenario cargo is creating (calling poll a bunch of times from separate threads, or something)?

…

On Thu, Oct 19, 2017 at 10:37 AM, Alex Crichton ***@***.***> wrote: I can't find anything on google that points to 10.9 specifically (everyone seems to claim 10.12 is broken) and all the "brokenness" appears to be in different ways? If we want to fix this we probably just want to have a fallback to select in cases like this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n1lXZsV_la_BREwPmkrKOubvvkDvks5st16pgaJpZM4P-peZ> .

alexcrichton · 2017-10-19T22:44:48Z

AFAIK the only "funky" thing here is that the fds are shared across multiple processes, but other than that there's not a whole lot going on here?

ivanceras · 2017-10-20T15:46:47Z

+1 on OSX Yosemite 10.10.3

cargo -V
cargo 0.23.0-nightly (e447ac7e9 2017-09-27)
rustc -V
rustc 1.22.0-nightly (4750c1ec0 2017-10-19)

panic error:

thread '<unnamed>' panicked at 'timeout in poll?', /Users/travis/.cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/jobserver-0.1.7/src/lib.rs:505:29

alexcrichton · 2017-10-20T17:29:09Z

I'm going to hope that rust-lang/jobserver-rs@400ce8c fixes this.

durka · 2017-10-20T17:34:46Z

Let me know if/how I can help test.

…

On Fri, Oct 20, 2017 at 1:29 PM, Alex Crichton ***@***.***> wrote: I'm going to hope that ***@***.*** <rust-lang/jobserver-rs@400ce8c> fixes this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3nzAyu9P-8grDVOnq0UeO5NcLIMgpks5suNhrgaJpZM4P-peZ> .

alexcrichton · 2017-10-20T17:35:55Z

Can you compile Cargo from source? If so, can you try compiling Cargo with that patch and see if it fixes things?

durka · 2017-10-20T17:38:45Z

OK, I'll let you know how that goes. Can I make rustup use a custom cargo and my installed rustc or how does that work?

…

On Fri, Oct 20, 2017 at 1:35 PM, Alex Crichton ***@***.***> wrote: Can you compile Cargo from source? If so, can you try compiling Cargo with that patch and see if it fixes things? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n002cxpU2iCU3fQ_WSc1DHFjspPMks5suNn9gaJpZM4P-peZ> .

alexcrichton · 2017-10-20T17:47:32Z

You'd probably just do something like rustup run beta path/to/target/release/cargo build and that should work ok

durka · 2017-10-20T17:59:27Z

When I do that it's somehow still running rustup's version of cargo... not sure what's going on.

…

On Fri, Oct 20, 2017 at 1:47 PM, Alex Crichton ***@***.***> wrote: You'd probably just do something like rustup run beta path/to/target/release/cargo build and that should work ok — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n-RthJi8XBUudaaLE-QuhbzgLAWXks5suNy2gaJpZM4P-peZ> .

durka · 2017-10-20T18:01:07Z

Oh no the problem is that librustc_trans also depends on jobserver :( so I need to build the whole compiler as well.

…

On Fri, Oct 20, 2017 at 1:59 PM, Alex Burka ***@***.***> wrote: When I do that it's somehow still running rustup's version of cargo... not sure what's going on. On Fri, Oct 20, 2017 at 1:47 PM, Alex Crichton ***@***.***> wrote: > You'd probably just do something like rustup run beta > path/to/target/release/cargo build and that should work ok > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#4643 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAC3n-RthJi8XBUudaaLE-QuhbzgLAWXks5suNy2gaJpZM4P-peZ> > . >

durka · 2017-10-20T19:06:28Z

@alexcrichton maybe this issue is in the wrong repo now, but I can't get rustc to build with that patch.

I'm running ./x.py build -i --stage 1 src/libtest. It fails while compiling stage1 test artifacts with "timeout in poll?" Fine. I can then run ./x.py build -i --stage 1 src/libtest -j 1 to complete the build.

I edited src/Cargo.toml to override jobserver with the git repo. Now it still fails during stage1 test artifacts, but attempting to continue with -j 1 yields

Building stage1 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
   Compiling std_unicode v0.0.0 (file:///Users/alex/Programming/rust/rust/src/libstd_unicode)
thread '<unnamed>' panicked at 'failed to shut down worker thread', /Users/alex/.cargo/git/checkouts/jobserver-rs-27359438239da508/400ce8c/src/lib.rs:641:16
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::begin_panic
   6: jobserver::imp::Helper::join
   7: <jobserver::HelperThread as core::ops::drop::Drop>::drop
   8: core::ptr::drop_in_place
   9: core::ptr::drop_in_place
error: Error during translation/LLVM phase.

error: Could not compile `std_unicode`.

alexcrichton · 2017-10-20T19:19:28Z

Ah oops I think I botched the patch. Forgot that EINTR needs to come out as an error and not get handled. Mind testing rust-lang/jobserver-rs@b3e6165?

durka · 2017-10-20T20:32:12Z

After rebuilding rustc and cargo with that patch it seems to be working!

…

On Fri, Oct 20, 2017 at 3:19 PM, Alex Crichton ***@***.***> wrote: Ah oops I think I botched the patch. Forgot that EINTR needs to come out as an error and not get handled. Mind testing alexcrichton/jobserver-rs@ b3e6165 <rust-lang/jobserver-rs@b3e6165> ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n8lxmBqozx5Hl6hZzbb2wNlMllFWks5suPJEgaJpZM4P-peZ> .

alexcrichton · 2017-10-20T21:49:57Z

Ok thanks for checking! I've pushed a new version of jobserver so all that's left is to send a PR to rust-lang/rust updating the crate

durka · 2017-10-20T22:18:40Z

And rust-lang/cargo, yeah? Are you doing them or shall I?

…

On Fri, Oct 20, 2017 at 5:49 PM, Alex Crichton ***@***.***> wrote: Ok thanks for checking! I've pushed a new version of jobserver so all that's left is to send a PR to rust-lang/rust updating the crate — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4643 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n50xpRD17ewYOEqXeAOP9V-X4mqNks5suRWGgaJpZM4P-peZ> .

alexcrichton · 2017-10-20T22:28:42Z

Nah rust-lang/cargo no longer has a lockfile, and feel free to do it!

ivanceras · 2017-10-21T19:27:00Z

A quick question: Will this fix ship on the next version of cargo? Is doing rustup self update && rustup update on the next nightly release enough to update cargo?

@alexcrichton

update jobserver version to work around macos bug Update `jobserver` crate to fix rust-lang/cargo#4643, a panic which can't happen according to `libc::poll`'s man page but was nevertheless reported on macOS 10.9 and 10.10. r? @alexcrichton

alexcrichton · 2017-10-23T15:45:24Z

@ivanceras this needed to land in rust-lang/rustc, which it now has, and now it should be in nightlies.

ivanceras · 2017-10-23T19:57:44Z

@alexcrichton I've just updated via rustup. The errors are now gone, it works great. Thanks!

durka mentioned this issue Oct 20, 2017

update jobserver version to work around macos bug rust-lang/rust#45423

Merged

bors closed this as completed in rust-lang/rust#45423 Oct 22, 2017

cargo panics with "timeout in poll?" error from jobserver #4643

cargo panics with "timeout in poll?" error from jobserver #4643

Comments

durka commented Oct 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

alexcrichton commented Oct 19, 2017

Uh oh!

durka commented Oct 19, 2017 via email

Uh oh!

alexcrichton commented Oct 19, 2017

Uh oh!

ivanceras commented Oct 20, 2017

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

durka commented Oct 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

durka commented Oct 20, 2017 via email

Uh oh!

alexcrichton commented Oct 20, 2017

Uh oh!

ivanceras commented Oct 21, 2017

Uh oh!

alexcrichton commented Oct 23, 2017

Uh oh!

ivanceras commented Oct 23, 2017

Uh oh!

durka commented Oct 19, 2017 •

edited

Loading

durka commented Oct 20, 2017 •

edited

Loading