CI logs should be split by build steps, or stages at the very least.

[eddyb]

Right now most of the build is under a single "run the build" CI step, which results in massive logs (e.g. #70468), even if for common tasks like looking at a failure, only a small part is relevant.

At the very least we could do something like this to split it up:

• ./x.py build --stage 0
• ./x.py build --stage 1
• ./x.py build
• ./x.py test

This would be trickier with distcheck, I suppose, not sure how relevant that is.

It might also be slightly slower, so we should measure it.

cc @rust-lang/infra

[kennytm]

One issue with splitting them into CI stages is that it requires uploading/download all the artifacts (the entire build/ directory), since each stage uses a brand new container.

Or we could use a different log file for each x.py stage, and print the log out in several CI jobs (but you lose the ability of real-time monitoring) 🤔

[Mark-Simulacrum]

Hm, I think that's for jobs, not steps though? We (for example) install the AWS tools in a separate step today and don't copy things around.

[pietroalbini]

That's surely possible, but I have no clue what the impact on our configuration would be (as we'd need different sets of steps).

[Mark-Simulacrum]

Yes, I think personally this would only be viable if GitHub provided a way to "start" a step dynamically -- similar to Travis' folders which were based on printing a message.