Cargo hits OOM when building many examples

[simply-silver]

Problem

When building examples on a large project with many of them, Cargo quickly runs through all available memory.

Steps

1. Clone a repository with many examples (e.g. Bevy with 142 examples)
2. Run cargo build --examples

Possible Solution(s)

It should be possible for Cargo to identify jobs that will consume high amounts of memory and ensure that other high-memory jobs don't occur at the same time.

Notes

No response

Version

cargo 1.67.0 (8ecd4f20a 2023-01-10)
release: 1.67.0
commit-hash: 8ecd4f20a9efb626975ac18a016d480dc7183d9b
commit-date: 2023-01-10
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.86.0-DEV (sys:0.4.59+curl-7.86.0 vendored ssl:OpenSSL/1.1.1q)
os: Arch Linux Rolling Release [64-bit]

[weihanglo]

Is that propositional to the number of examples? How large is the RAM of your machine?
I'd say that it's a bit hard for Cargo to predict memory usage beforehand. Profile data could be persisted but Cargo currently doesn't have such a persistence mechanism.

It would be helpful if you can share memory profiling data here. There are several tools out there such as heaptrack. Here is also a list of profiling tools.

[ThomasWaldmann]

Maybe related (not with examples, but a real project): pyca/cryptography#8640

[nathaniel-brough]

I've run into a similar issue with building surrealdb on github actions. Where it builds perfectly fine on my local machine. Without some detailed heap profiling I eyeballed the memory usage of the docker-container I was using to build on my local machine and it was peaking at about 6.7GB when compiling the surrealdb package.

[epage]

Closing in favor of #12912 so we keep the conversation in one place.