Skip to content

A heisenbug in aarch64-linux-android CI #2081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
workingjubilee opened this issue Feb 25, 2021 · 1 comment
Open

A heisenbug in aarch64-linux-android CI #2081

workingjubilee opened this issue Feb 25, 2021 · 1 comment
Labels
A-CI Area: CI-related items C-bug Category: bug E-medium E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. O-android O-arm
Milestone

Comments

@workingjubilee
Copy link
Member

workingjubilee commented Feb 25, 2021

CI for that target is currently disabled due to this issue.

Hello, traveler.

I have stories to tell of strange bugs from unseen lands. Monsters which flicker in and out of sight, vanishing in the blink of an eye, hidden in virtual domains that themselves coalesce into being and then topple back into sand.

This primarily affects the aarch64-linux-android CI but it also affects the sparc-unknown-linux-gnu CI.
This was found at least as far back at #1851.
Several error logs are quoted into #2078.
The associated errors seemed to occur more often during the middle of the day rather than at night, so it may be an occurrence of a "noisy neighbor" problem in the virtual environment. It may also be an issue with QEMU.

This happens near the the cmsg tests, but can also happen during errqueue tests, the primary messages are

cmsg

stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:45:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--test cmsg'

errqueue

stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/git.colasdn.top-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:46:9
stack backtrace:
   0: std::panicking::begin_panic
   1: runtest::main::{{closure}}
   2: core::option::Option<T>::unwrap_or_else
   3: runtest::main
   4: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--test errqueue'

In 5c07fcf we decided to add, and for the moment keep, RUST_BACKTRACE=1 at appropriate points in CI so as to hopefully surface a more useful backtrace when this triggers again. In fcae5a7 we decided to cut aarch64-linux-android out of CI.

But we could do no more than place that warding to blockade that cyclopean archway. The monster still lives, and in time, it may come again, between the blinks of our strange existences.

@maxbla
Copy link
Contributor

maxbla commented Feb 26, 2021

Link to that line in getopts. I'm not at all familiar with this crate's tests and looked through, but can't find which CI tests depend on getopts.

@JohnTitor JohnTitor added the A-CI Area: CI-related items label Feb 26, 2021
@tgross35 tgross35 added the E-medium E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. label Aug 29, 2024
@tgross35 tgross35 added this to the 1.x milestone Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-CI Area: CI-related items C-bug Category: bug E-medium E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. O-android O-arm
Projects
None yet
Development

No branches or pull requests

4 participants