cargo fetch can't be parallelized on nightly anymore

[pietroalbini]

Problem

It used to be possible to run multiple cargo fetch in parallel inside the same cargo home, but it's not possible anymore on the latest nightly: the following fetch invocations just wait for the package cache lock until the other fetches finish.

This is bad for Crater since it slows down fetching deps to 1/8.

Steps

#!/bin/bash
set +euo pipefail

if [[ $# -ne 1 ]]; then
    echo "usage: $0 <toolchain-name>"
    exit 1
fi
export RUSTUP_TOOLCHAIN="$1"

# Use a temporary cargo home so the user-wide one is not messed up
export CARGO_HOME=tmp-cargo-home

set -x

# Create a simple repro crate which depends on lazy_static
cargo init --bin repro
cd repro
echo 'reqwest = "*"' >> Cargo.toml

# Generate the lockfile and fetch the dependencies using it
cargo generate-lockfile
cargo fetch --locked >/dev/null 2>&1 &
sleep 0.1
cargo fetch --locked

• bash repro.sh stable for the correct behavior
• bash repro.sh nightly for the wrong behavior

Possible Solution(s)

After talking to @Eh2406 the issue seems to be caused by #6880.

cc @alexcrichton

[alexcrichton]

This is sort of intentional in the sense that previously Cargo attempted to do fine-grained locking but it became both a performance issue and a correctness burden on Cargo for a use case that very very rarely came up in practice. I believe #6880 is indeed the cause for this though.

[epage]

Considering the trade off involved here and the small scope of what was slowed down (while instead other cases were sped up), I'm not seeing us reverting to the old locking scheme. I'm also not seeing a viable alternative. With that in mind, I'm going to go ahead and close this.