miri component gets shipped even though the tests fail

[RalfJung]

Looking at https://rust-lang.github.io/rustup-components-history/, it seems like Miri is getting shipped as a rustup component even though the tests fail. Unfortunately, Miri is currently entirely usable, so it'd probably be better if it did not get shipped in this state. Is that something we can do?

Cc @oli-obk

[oli-obk]

maybe we can make the miri toolstate always go to build-fail even if it technically is just test-fail

[RalfJung]

@rust-lang/infra what would be the best place to change this? I think I'd prefer Miri not to be shipped at all when the tests fail; I'd rather people stick to an old-but-working Miri than to a new-but-broken one.

I found https://github.com/rust-lang/rust/blob/master/src/ci/docker/x86_64-gnu-tools/checktools.sh but that does not seem to decide if a component gets built for distribution or not.

[oli-obk]

rust/src/bootstrap/dist.rs

Lines 1368 to 1369 in 6312b89

	// We expect miri to build, because we've exited this step above if tool
	// state for miri isn't testing.

seems to suggest that it is supposed to work as you describe that it should be working. Since it obviously doesn't and I can't see how that comment could be true from looking at the source, I'm guessing this is some refactoring fallout. I'll dig a little into git blame

[oli-obk]

I found it. The comment was added for rls in 686c101 but then the code making the comment true was removed later

[RalfJung]

See #61656 (comment) for why this is hard to fix: the dist builders don't even run the tool tests, so they cannot know if the tests pass. :/

Also, that's probably for the better actually, as the "tools" builder sets --cfg miri to be more Miri-friendly and we don't want that in the release artifacts.

[RalfJung]

One (crazy?) thing we could so is have the dist builders query the toolstate repo to figure out if the tests are passing...

[kennytm]

That will either retrieve an outdated toolstate or will need to block until the toolstate is uploaded ...

[RalfJung]

Without blocking it would lag behind by 1 merged PR, yes.

[mati865]

@RalfJung would it be possible to add some kind of build assert so Miri fails on dist builders when Rust internals change too much?

[RalfJung]

Well dist builders already know when Miri fails to build because internals change too much. The issue is things that make Miri still compile but break the test suite.

[RalfJung]

@kennytm would it be possible to at least change the logic that opens issues such that for Miri, it opens an issue like #60533 when the tests fail? I could also try to implement that if you point me to the right place.

So far we have had zero cases of Miri tests spuriously failing.

[mati865]

That should be easy to do:

rust/src/tools/publish_toolstate.py

Line 152 in 0e4a56b

if build_failed: