Substantial debug-test-optimized-depedencies Performance Regression in 1.59

[TheBlueMatt]

We were trying to change our Cargo.toml to

[profile.dev.package."*"]
opt-level = 2

to get opt on our dependencies during test but not on the crate being tested. In rustc 1.58.1, this is a performance improvement in the generated test suite. On rustc 1.59.0 this is a very substantial regression in total runtime of tests (~2 minutes CPU time on 1.58.1 to 18 minutes CPU time, not including compile time). This effect is robust with different opt-levels (swapping the 1s and 2s above) and between Debian rustc and rustup rustc.

The increase in CPU time seems to almost all be in std/core, perf on the regressed binary for a single test (similar results for all tests) shows the following, but core itself doesn't show up at all in perf on 1.58.1.

+   26.40%    24.37%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] core::ptr::mut_ptr::<impl *mut T>::is_null
+   25.79%    23.79%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] <core::slice::iter::IterMut<T> as core::iter::traits::iterator::Iterator>::next
+   23.12%    21.85%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] core::ptr::mut_ptr::<impl *mut T>::guaranteed_eq
+   11.31%    10.16%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] std::io::readbuf::ReadBuf::initialize_unfilled_to
+    9.80%     6.16%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] core::ptr::non_null::NonNull<T>::as_ptr
+    8.65%     6.89%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] core::ptr::metadata::from_raw_parts_mut
+    2.46%     2.45%  ln::functional_  lightning-8c1dd3bce02d4ddc   [.] core::ptr::non_null::NonNull<T>::new_unchecked

This effect is also robust against -Zbuild-std=panic_abort (and ps aux confirms that core is getting built with opt-level=2) and even RUSTFLAGS="-C prefer-dynamic" (which ldd confirms dynamically links libstd+libtest).

The PR to do this on our repo is at lightningdevkit/rust-lightning#1497 which you should be able to simply check out locally and cargo test with 1.58 and 1.59 (well, run it twice to skip the compile part of the test) to see the regression. Specific numbers at #97460 (comment).

For those reading the comments there was some confusion about the cargo 1.57 change on the test profile, you can skip to #97460 (comment)

@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged

[wesleywiser]

[profile.dev]
opt-level = 1

Sets the rustc -Copt-level=1 flag when compiling your dependencies as well as your crate.

[profile.dev.package."*"]
opt-level = 2

Sets the rustc -Copt-level=2 flag when compiling only your dependencies, leaving your crate built with -Copt-level=0.

[TheBlueMatt]

Sets the rustc -Copt-level=1 flag when compiling your dependencies as well as your crate.

So this changed in cargo/rustc 1.59?

[TheBlueMatt]

Sets the rustc -Copt-level=1 flag when compiling your dependencies as well as your crate.

Also, this doesn't match ps aux output during build - on 1.58.1 and 1.59 I show no opt-level argument passed to rustc at all for the --test build.

[TheBlueMatt]

So this changed in cargo/rustc 1.59?

To answer my own question, yes, this changed in 1.59, dev now passes opt-level=1 to rustc on 1.59 on the --test build. That said, it doesn't address the point here - opt-level=0 (looking at rustc calls, at least) on 1.58 led to substantially more performant code.

[TheBlueMatt]

(FWIW this does match my observation that rustc 1.59 [dev] opt-level=1 runs much faster than 1.58 [dev] opt-level=1, but it seems there's no option to get back to the performance/compile-time tradeoffs we had at 1.58)