Rustup doesn't handle HTTP 416 errors when downloading toolchains

[pietroalbini]

Problem

While setting up a Crater agent the container running Crater was killed for reasons in the middle of a rustup download. After starting the container again rustup kept failing with this error message:

info: downloading component 'rust-std'
error: component download failed for rust-std-x86_64-unknown-linux-gnu
error: caused by: could not download file from 'https://static.rust-lang.org/dist/2019-09-26/rust-std-1.38.0-x86_64-unknown-linux-gnu.tar.xz' to '/workspace/rustup-home/downloads/51b334337804baeff7524e5496d396f254894d7529860d236975e9ed8fcca371.partial'
error: caused by: http request returned an unsuccessful status code: 416

Steps

Note that these steps didn't cause the original bug, but they reproduce the same symptoms.

1. Remove all partial downloads:

rm -f ~/.rustup/downloads/*.partial

2. Install a toolchain, but kill rustup in the middle of a download.
3. Increase the size of the only .partial file to 1 GB, way above the total size of the file:

fallocate -l 1G ~/.rustup/downloads/*.partial

4. Try to reinstall the same toolchain, it will fail with:

error: component download failed for rust-std-x86_64-unknown-linux-gnu
error: caused by: could not download file from 'https://static.rust-lang.org/dist/2019-01-01/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz' to '/home/pietro/.rustup/downloads/63e41b2768326b80fbf9bdef657bb3c717319538e75b52dce305bc4363dce1f6.partial'
error: caused by: http request returned an unsuccessful status code: 416

Possible Solution(s)

The HTTP 416 status code means the content-range requested to the server was not satisfiable, which in our case means a corrupted (or overly large like in my repro steps) partial file. Rustup could handle this better by ignoring the partial and starting to download from scratch when it receives a 416 status code.

Notes

Output of rustup --version:

rustup 1.20.2 (13979c968 2019-10-16)

Output of rustup show: not relevant

[kinnison]

Related to #1854 and #1889

[kinnison]

Hi @pietroalbini

We've merged some code to remove spurious or broken .partial files and to instruct the user to try again. Could you possibly have a go with a current master of rustup (I can show you how if you don't know) and see if this behaves more usefully for you now?

[pietroalbini]

Tested this. The partial file is actually removed after the build finishes, but you have to re-run rustup again manually to actually download the release. It'd be awesome if rustup could just retry the request on its own.

[jyn514]

I had this same error in what seems to be an old version of rustup. Running rustup self update does nothing, though. When will the next version of rustup be published?

$ rustup install stable
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2019-11-07, rust version 1.39.0 (4560ea788 2019-11-04)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
error: component download failed for rust-std-x86_64-unknown-linux-gnu
error: caused by: could not download file from 'https://static.rust-lang.org/dist/2019-11-07/rust-std-1.39.0-x86_64-unknown-linux-gnu.tar.xz' to '/home/joshua/.rustup/downloads/b629952ec9e3e750137d7b8a922de145c97c1dc372dd616e86e4e501da13910b.partial'
error: caused by: http request returned an unsuccessful status code: 416
$ rustup --version
rustup 1.20.2 (13979c968 2019-10-16)
$ rustup self update
info: checking for self-updates
$ rustup --version
rustup 1.20.2 (13979c968 2019-10-16)

[kinnison]

@jyn514 Yes, this is slated to be fixed by 1.21.0 (see the milestone) but that's not released yet (see the milestone) :D