ICE when include_bytes-ing ~>1GB of data in lib.rs #103607
Labels
C-bug
Category: This is a bug.
I-ICE
Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️
S-has-mcve
Status: A Minimal Complete and Verifiable Example has been found for this issue
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
Comments
Heliozoa
added
C-bug
|
The crash happens in the metadata encoding here: rust/compiler/rustc_metadata/src/rmeta/table.rs Lines 233 to 234 in 898f463 This code converts MRE is #![crate_type = "lib"]
const _FILE: &[u8] = include_bytes!("./file");create a Not honestly sure what is a proper fix here. |
|
@rustbot label +S-bug-has-mcve |
rustbot
added
the
S-has-mcve
|
Is there a workaround for this or some other way to include the file as bytes in the code? |
|
@soupslurpr not that I'm aware of, I'm afraid. Although as a bit of an update, @saethlin is currently working on lifting the (we also need to check why it takes so long to include big files and possibly try to optimize the code there, but it's another thing) |
This was referenced Aug 27, 2023
|
You may be able to work around this by moving the large There's also a performance/memory issue that happens when you get to there though. I'm now working on that too, I'll link a PR to this issue if I come up with a fix. |
|
OK cool, I'll just wait for it to get merged since I'm doing something else right now. Thanks |
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this issue
Aug 31, 2023
Adapt table sizes to the contents This is an implementation of rust-lang/compiler-team#666 The objective of this PR is to permit the rmeta format to accommodate larger crates that need offsets larger than a `u32` can store without compromising performance for crates that do not need such range. The second commit is a number of tiny optimization opportunities I noticed while looking at perf recordings of the first commit. The rmeta tables need to have fixed-size elements to permit lazy random access. But the size only needs to be fixed _per table_, not per element type. This PR adds another `usize` to the table header which indicates the table element size. As each element of a table is set, we keep track of the widest encoded table value, then don't bother encoding all the unused trailing bytes on each value. When decoding table elements, we copy them to a full-width array if they are not already full-width. `LazyArray` needs some special treatment. Most other values that are encoded in tables are indexes or offsets, and those tend to be small so we get to drop a lot of zero bytes off the end. But `LazyArray` encodes _two_ small values in a fixed-width table element: A position of the table and the length of the table. The treatment described above could trim zero bytes off the table length, but any nonzero length shields the position bytes from the optimization. To improve this, we interleave the bytes of position and length. This change is responsible for about half of the crate metadata win on many crates. Fixes rust-lang/rust#112934 (probably) Fixes rust-lang/rust#103607
lnicola
pushed a commit
to lnicola/rust-analyzer
that referenced
this issue
Apr 7, 2024
Adapt table sizes to the contents This is an implementation of rust-lang/compiler-team#666 The objective of this PR is to permit the rmeta format to accommodate larger crates that need offsets larger than a `u32` can store without compromising performance for crates that do not need such range. The second commit is a number of tiny optimization opportunities I noticed while looking at perf recordings of the first commit. The rmeta tables need to have fixed-size elements to permit lazy random access. But the size only needs to be fixed _per table_, not per element type. This PR adds another `usize` to the table header which indicates the table element size. As each element of a table is set, we keep track of the widest encoded table value, then don't bother encoding all the unused trailing bytes on each value. When decoding table elements, we copy them to a full-width array if they are not already full-width. `LazyArray` needs some special treatment. Most other values that are encoded in tables are indexes or offsets, and those tend to be small so we get to drop a lot of zero bytes off the end. But `LazyArray` encodes _two_ small values in a fixed-width table element: A position of the table and the length of the table. The treatment described above could trim zero bytes off the table length, but any nonzero length shields the position bytes from the optimization. To improve this, we interleave the bytes of position and length. This change is responsible for about half of the crate metadata win on many crates. Fixes rust-lang/rust#112934 (probably) Fixes rust-lang/rust#103607
RalfJung
pushed a commit
to RalfJung/rust-analyzer
that referenced
this issue
Apr 27, 2024
Adapt table sizes to the contents This is an implementation of rust-lang/compiler-team#666 The objective of this PR is to permit the rmeta format to accommodate larger crates that need offsets larger than a `u32` can store without compromising performance for crates that do not need such range. The second commit is a number of tiny optimization opportunities I noticed while looking at perf recordings of the first commit. The rmeta tables need to have fixed-size elements to permit lazy random access. But the size only needs to be fixed _per table_, not per element type. This PR adds another `usize` to the table header which indicates the table element size. As each element of a table is set, we keep track of the widest encoded table value, then don't bother encoding all the unused trailing bytes on each value. When decoding table elements, we copy them to a full-width array if they are not already full-width. `LazyArray` needs some special treatment. Most other values that are encoded in tables are indexes or offsets, and those tend to be small so we get to drop a lot of zero bytes off the end. But `LazyArray` encodes _two_ small values in a fixed-width table element: A position of the table and the length of the table. The treatment described above could trim zero bytes off the table length, but any nonzero length shields the position bytes from the optimization. To improve this, we interleave the bytes of position and length. This change is responsible for about half of the crate metadata win on many crates. Fixes rust-lang/rust#112934 (probably) Fixes rust-lang/rust#103607
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code
dd if=/dev/random of=file bs=1G count=1I was not able to reproduce the crash when I moved the code to main.rs.
Including one ~750MB file did not cause a crash, but including both a ~750MB file and a ~250MB file did.
Meta
Was able to reproduce on stable and nightly
rustc --version --verbose:rustc +nightly --version --verbose:Error output
When running
cargo (+nightly) checkBacktrace
The text was updated successfully, but these errors were encountered: