-
Notifications
You must be signed in to change notification settings - Fork 480
Rust application much slower when built with rules_rust than with Cargo #3407
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist. |
I've been sorting, diffing, comparing the symbol tables with |
Got it, I'd be curious to see some examples of short functions where there's a diff (something innocuous in terms of confidentiality). |
@lgarbarini I haven't had a chance to try that tool yet but it does seem like the slowdown is evenly spread across two particular crates that really need to be fast and get inlined/LTO'd a lot. I don't know what elf_diff would show me that nm, samply, and bloaty haven't already, I've been comparing the symbol tables. |
I think my message was unclear, completely agree that objdump, nm, samply, and bloaty have you covered, no need for elf_diff if you're doing it manually. If the manual diff is exclusively inlined functions then I'll poke around some more, but if the diff contains other changes to other instructions outside of inlining, I would be curious to see a few excerpted sections from objdump. |
While we're waiting for elf_diff to finish, here's how I've been comparing the symbols:
The differences just aren't significant at least from what I can see from
Is an example of the output for the middle commands in between the diffs and the uniq counts. |
Someone in another thread raised it so I'll say it explicitly here: I have profiled this over and over. The slowdown is spread across the entire call graph of the 1 or 2 main crates that impact performance, there are no hot-spots, just deeply rooted parts of the call graph that get wider as a whole. I've been using samply to profile and it seems to be capturing everything but I'm running with cachegrind right now to try to get a more complete profile. If there was a hotspot I'd have ideas for what to try other than fiddling with |
Wow, this is quite something. Just want to follow along to see what comes up in the investigation. |
Also, have you verified that the resulting binaries are actually statically linked? I have reported god knows how many issues related to static linking with rules_rust and ultimately given up. |
@marvin-hansen There's only so much I can say about this which is related to why an MRE is so hard, but a few things:
My main update at this point is that I'm no longer able to reproduce the original Cargo baseline benchmark performance on Linux…with Cargo. Even if I backtrack to older commits. I can still repro it on Mac + Cargo. It's a macro-benchmark that involves a very thick stack of technology, I can't really make an MRE that meaningfully captures this. I will scrutinize my micro-benchmarks and see if any of them are evincing the same slow-down factor. If such a thing exists, then I will extract an MRE, but there's always a chance that whatever fixes the micro-benchmark doesn't fix anything for the big boy. |
@lgarbarini I looked at the elf_diff results this morning, I saw some false deleted/added where it was the same symbol with the same asm read-out getting flagged for unknown reasons. Nothing in particular stood out but I haven't had time to scrutinize it carefully. I'm going to focus on an MRE and shrinking the repro because there's just too much stuff happening in this benchmark and I need to isolate variables. |
Yeah I'm still having this perf issue with the Bazel-built version of my program. I managed to get some unrelated issues fixed and got back to the original baseline performance for the Cargo and Bazel versions, same situation as before. As near as I can tell it's like the usual LTO and LLVM optimizations aren't happening. An example is that we have an empty inline asm! hack as a branch hint, I'm pretty sure it's not getting optimized out in the Bazel version but I haven't confirmed yet. We rely on LLVM to do that. Another thing I'll note here is that the problem libm thing: rust-lang/libm#545 Update: I replaced the Update2: I can only repro this on macOS, not Linux. This is increasingly looking like something weird with macOS or ld64, but I'm not sure. |
Nope, it's repro'ing with Cargo inside of Docker + OrbStack + Ubuntu 24.10 + aarch64 on my M3 Pro Mac. Suspiciously faster execution:
Slower execution:
My local test bench for Linux is a 9800X3D + RTX 5090 desktop that I built recently, the benchmark is single-threaded and the single-threaded performance of the M3 Pro and 9800X3D are generally very close which is convenient for my purposes. The commonality here so far seems to be cargo + aarch64 rather than macOS/Mac toolchain. Update: there seems to be something specifically about letting |
Optimization flag perf impact: Docker, OrbStack, M3 Mac, Bazel: (these are sequential performance steps/changes)
Docker, OrbStack, M3 Mac, Cargo: (sequential again)
When I disabled |
After writing a sed script for redacting some sensitive crate names, here are the full final build commands for the benchmark binary from cargoCARGO=/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/bin/cargo
CARGO_BIN_EXE_app-nm=/app/target/release/app-nm
CARGO_CRATE_NAME=my_benchmark
CARGO_MANIFEST_DIR=/app/app-nm
CARGO_MANIFEST_PATH=/app/app-nm/Cargo.toml
CARGO_PKG_AUTHORS=''
CARGO_PKG_DESCRIPTION=''
CARGO_PKG_HOMEPAGE=''
CARGO_PKG_LICENSE=''
CARGO_PKG_LICENSE_FILE=''
CARGO_PKG_NAME=app-nm
CARGO_PKG_README=README.md
CARGO_PKG_REPOSITORY=''
CARGO_PKG_RUST_VERSION=''
CARGO_PKG_VERSION=0.1.0
CARGO_PKG_VERSION_MAJOR=0
CARGO_PKG_VERSION_MINOR=1
CARGO_PKG_VERSION_PATCH=0
CARGO_PKG_VERSION_PRE=''
CARGO_PRIMARY_PACKAGE=1
CARGO_SBOM_PATH=''
CARGO_TARGET_TMPDIR=/app/target/tmp
LD_LIBRARY_PATH='/app/target/release/deps:/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/lib'
/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/bin/rustc
--crate-name
my_benchmark
--edition=2021
app-nm/benches/my_benchmark.rs
--error-format=json
--json=diagnostic-rendered-ansi,artifacts,future-incompat
--diagnostic-width=106
--emit=dep-info,link
-C
opt-level=3
-C
lto=thin
-C
codegen-units=1
-C
debuginfo=1
--cfg
test
--check-cfg
'cfg(docsrs,test)'
--check-cfg
'cfg(feature,
values())'
-C
metadata=65b29445035900e0
-C
extra-filename=-5a438e8f8ec6ae3d
--out-dir
/app/target/release/deps
-L
dependency=/app/target/release/deps
--extern
anyhow=/app/target/release/deps/libanyhow-b1edd40a0e594b17.rlib
--extern
bs58=/app/target/release/deps/libbs58-fae0e5699db31778.rlib
--extern
nm-ch=/app/target/release/deps/libnm-ch-68f6ac10eeca7a26.rlib
--extern
chrono=/app/target/release/deps/libchrono-6c2368740715bfe1.rlib
--extern
clap=/app/target/release/deps/libclap-9acdcde543eb8016.rlib
--extern
criterion=/app/target/release/deps/libcriterion-f607b0ecd2564939.rlib
--extern
nm_c=/app/target/release/deps/libnm-c-0bb82d835be9c8cf.rlib
--extern
either=/app/target/release/deps/libeither-f430a192a1c2d6c9.rlib
--extern
getrandom=/app/target/release/deps/libgetrandom-ee380435cb066fcd.rlib
--extern
gnort=/app/target/release/deps/libgnort-330afad64b885c9e.rlib
--extern
ibig=/app/target/release/deps/libibig-eb5fc25037ace091.rlib
--extern
artifacts=/app/target/release/deps/libartifacts-df5162a314165126.rlib
--extern
libc=/app/target/release/deps/liblibc-cf0d128ff4389ed5.rlib
--extern
l2=/app/target/release/deps/libl2-f79f90140ac6f04a.rlib
--extern
nmnc=/app/target/release/deps/libnmnc-47a721e54fbfdca0.rlib
--extern
nmnc_l2=/app/target/release/deps/libnmnc_l2-cc7422f92be1ebd4.rlib
--extern
nmnc_m=/app/target/release/deps/libnmnc_m-b09243d00dce314e.rlib
--extern
num_cpus=/app/target/release/deps/libnum_cpus-1411bfdf63d9102e.rlib
--extern
perf_event=/app/target/release/deps/libperf_event-773f2ee67ac5840f.rlib
--extern
pin_project_lite=/app/target/release/deps/libpin_project_lite-d72327f10b8ee167.rlib
--extern
rand=/app/target/release/deps/librand-5057525688919945.rlib
--extern
reqwest=/app/target/release/deps/libreqwest-f599cef6ebc0021c.rlib
--extern
serde=/app/target/release/deps/libserde-68bb54fbdf39400f.rlib
--extern
serde_json=/app/target/release/deps/libserde_json-ccacf4f8583e1327.rlib
--extern
nms=/app/target/release/deps/libnms-56e9cf15c14b887b.rlib
--extern
nms_macros=/app/target/release/deps/libnms_macros-bc968a627bfa56a4.so
--extern
tempfile=/app/target/release/deps/libtempfile-c8c38e0b986b3748.rlib
--extern
testcontainers=/app/target/release/deps/libtestcontainers-d4eb9115a94b91b1.rlib
--extern
thiserror=/app/target/release/deps/libthiserror-df954635be403dad.rlib
--extern
tokio=/app/target/release/deps/libtokio-9dc8c59b1c711c7d.rlib
--extern
tokio_util=/app/target/release/deps/libtokio_util-7fd2c8732417e306.rlib
--extern
tracing=/app/target/release/deps/libtracing-c0e09c928a721c66.rlib
--extern
tracing_subscriber=/app/target/release/deps/libtracing_subscriber-967fe0e0afece883.rlib
--extern
utilities=/app/target/release/deps/libutilities-312805f8dc6cf16a.rlib
--extern
a2=/app/target/release/deps/liba2-d95cb3346f5f17ca.rlib
-L
native=/app/target/release/build/blake3-319ac08c6f4e13fb/out
-L
native=/app/target/release/build/ring-bb137da75991c7b4/out
-L
native=/app/target/release/build/ring-6e5ba74e6f60ab19/out
-L
native=/app/target/release/build/secp256k1-sys-4b18d0f5e952cbe1/out bazelnc-m/benches/my_benchmark.rs
--crate-name=my_benchmark
--crate-type=bin
--error-format=human
--out-dir=bazel-out/aarch64-opt/bin/nc-m
--codegen=opt-level=3
--codegen=debuginfo=0
--codegen=strip=debuginfo
--remap-path-prefix=${pwd}=
--emit=link=bazel-out/aarch64-opt/bin/nc-m/my_benchmark
--emit=dep-info
--color=always
--target=aarch64-unknown-linux-gnu
-L
bazel-out/aarch64-opt/bin/external/rules_rust++rust+rust_linux_aarch64__aarch64-unknown-linux-gnu__nightly_tools/rust_toolchain/lib/rustlib/aarch64-unknown-linux-gnu/lib
--edition=2021
-Clto=thin
--codegen=linker=external/toolchains_llvm++llvm+llvm_toolchain/bin/cc_wrapper.sh
--codegen=link-arg=--target=aarch64-unknown-linux-gnu
--codegen=link-arg=-no-canonical-prefixes
--codegen=link-arg=-lm
--codegen=link-arg=-fuse-ld=lld
--codegen=link-arg=-Wl,--build-id=md5
--codegen=link-arg=-Wl,--hash-style=gnu
--codegen=link-arg=-Wl,-z,relro,-z,now
--codegen=link-arg=-l:libc++.a
--codegen=link-arg=-l:libc++abi.a
--codegen=link-arg=-l:libunwind.a
--codegen=link-arg=-lpthread
--codegen=link-arg=-ldl
--codegen=link-arg=-rtlib=compiler-rt
--codegen=link-arg=-Wl,--gc-sections
--extern=nc_m=bazel-out/aarch64-opt/bin/nc-m/libnc_m-3221654267.rlib
--extern=ks=bazel-out/aarch64-opt/bin/ks/libks-2401207873.rlib
--extern=c=bazel-out/aarch64-opt/bin/na/c/libc-401854362.rlib
--extern=nc=bazel-out/aarch64-opt/bin/nc/libnc-2793115565.rlib
--extern=s=bazel-out/aarch64-opt/bin/s/rust/s/libs-262915972.rlib
--extern=a2=bazel-out/aarch64-opt/bin/a2/liba2-894933967.rlib
--extern=bs58=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bs58-0.5.1/libbs58-1951919875.rlib
--extern=clap=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap-4.5.37/libclap-186282177.rlib
--extern=either=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__either-1.15.0/libeither-1753065134.rlib
--extern=getrandom=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.2.15/libgetrandom-249734380.rlib
--extern=gnort=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__gnort-0.1.2/libgnort-1317023791.rlib
--extern=libc=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libc-0.2.172/liblibc-2400173565.rlib
--extern=libp2p=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-0.55.0/liblibp2p-1954658880.rlib
--extern=num_cpus=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_cpus-1.16.0/libnum_cpus-1232792777.rlib
--extern=pin_project_lite=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-lite-0.2.16/libpin_project_lite-2218591079.rlib
--extern=rand=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.8.5/librand-707188286.rlib
--extern=thiserror=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-2.0.12/libthiserror-2502407963.rlib
--extern=tokio=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-1.44.2/libtokio-2604446065.rlib
--extern=tokio_util=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-util-0.7.14/libtokio_util-50890834.rlib
--extern=tracing=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-0.1.41/libtracing-2194975523.rlib
--extern=tracing_subscriber=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-subscriber-0.3.19/libtracing_subscriber-324201358.rlib
--extern=anyhow=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anyhow-1.0.98/libanyhow-787701661.rlib
--extern=chrono=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__chrono-0.4.40/libchrono-937817766.rlib
--extern=criterion=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-0.5.1/libcriterion-161828024.rlib
--extern=reqwest=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__reqwest-0.12.15/libreqwest-737296596.rlib
--extern=serde=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde-1.0.219/libserde-498884776.rlib
--extern=serde_json=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_json-1.0.140/libserde_json-927511813.rlib
--extern=tempfile=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tempfile-3.19.1/libtempfile-1663650551.rlib
--extern=testcontainers=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__testcontainers-0.23.3/libtestcontainers-1979583287.rlib
--extern=perf_event=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-0.4.8/libperf_event-898811581.rlib
-Ldependency=bazel-out/aarch64-opt/bin/ks
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/assert_no_alloc
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cfg-if-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-traits-0.2.19
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerocopy-0.8.24
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ppv-lite86-0.2.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libc-0.2.172
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.2.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_core-0.6.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_chacha-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.8.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__static_assertions-1.1.0
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/ibig
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/murmur3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__typenum-1.18.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__generic-array-0.14.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crypto-common-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inout-0.1.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cipher-0.4.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cpufeatures-0.2.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aes-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aead-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dbl-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__block-buffer-0.10.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__subtle-2.6.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__digest-0.10.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cmac-0.7.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ctr-0.9.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zeroize_derive-1.4.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zeroize-1.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aes-siv-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__curve25519-dalek-4.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signature-2.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ed25519-2.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha2-0.10.8
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ed25519-dalek-2.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha1-0.10.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__x25519-dalek-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/s_crypto
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__funty-2.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__radium-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tap-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__wyz-0.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitvec-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__either-1.15.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__intmap-3.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__json-0.12.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lazy_static-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memmap2-0.9.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-registry-1.4.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-0.3.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__slotmap-1.0.7
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__thiserror-impl-2.0.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-2.0.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__log-0.4.27
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-lite-0.2.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__once_cell-1.21.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-core-0.1.33
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tracing-attributes-0.1.28
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-0.1.41
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/s/rust/s_macros
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__num-derive-0.4.2
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/s
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anyhow-1.0.98
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_derive-1.0.219
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde-1.0.219
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bytes-1.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-core-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fnv-1.0.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__itoa-1.0.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-1.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-body-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-body-util-0.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mime-0.3.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sync_wrapper-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-layer-0.3.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-service-0.3.3
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__rustversion-1.0.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-core-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__percent-encoding-2.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__form_urlencoded-1.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-sink-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-channel-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-io-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-task-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memchr-2.7.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-utils-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__slab-0.4.9
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__futures-macro-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-util-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__atomic-waker-1.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__equivalent-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__allocator-api2-0.2.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foldhash-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.15.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__indexmap-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mio-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__scopeguard-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lock_api-0.4.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__smallvec-1.15.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parking_lot_core-0.9.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parking_lot-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__socket2-0.5.9
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tokio-macros-2.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-1.44.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-util-0.7.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__h2-0.4.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__httparse-1.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__httpdate-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__try-lock-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__want-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-1.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-util-0.1.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchit-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ryu-1.0.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_json-1.0.140
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_path_to_error-0.1.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_urlencoded-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-0.8.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unty-0.0.4
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__bincode_derive-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bincode-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__arrayref-0.3.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__arrayvec-0.7.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__constant_time_eq-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__blake3-1.5.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__byteorder-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__iana-time-zone-0.1.63
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__chrono-0.4.40
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-1.0.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf8parse-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-parse-0.2.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-query-1.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__colorchoice-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__is_terminal_polyfill-1.70.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstream-0.6.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap_lex-0.7.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__strsim-0.11.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap_builder-4.5.37
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__clap_derive-4.5.32
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap-4.5.37
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__option-ext-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dirs-sys-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dirs-6.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_cpus-1.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-executor-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-utils-0.8.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.14.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dashmap-6.1.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__derive_more-impl-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__derive_more-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_core-0.9.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_chacha-0.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__retry-2.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dogstatsd-0.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-timer-3.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__no-std-compat-0.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nonzero_ext-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__portable-atomic-1.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quanta-0.12.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__spinning_top-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__web-time-1.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__governor-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__maplit-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__gnort-0.1.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__thiserror-impl-1.0.69
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-1.0.69
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-0.27.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.22.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__stable_deref_trait-1.2.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zerofrom-derive-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerofrom-0.1.6
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__yoke-derive-0.7.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yoke-0.7.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zerovec-derive-0.10.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerovec-0.10.4
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__displaydoc-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_collections-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_normalizer_data-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__litemap-0.7.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinystr-0.7.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__writeable-0.5.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid_transform_data-1.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__icu_provider_macros-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_provider-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid_transform-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_properties_data-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_properties-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf16_iter-1.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf8_iter-1.0.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__write16-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_normalizer-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__idna_adapter-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__idna-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__url-2.5.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__encoding_rs-0.8.35
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitflags-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foreign-types-shared-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foreign-types-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-sys-0.9.107
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__openssl-macros-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-0.10.72
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-probe-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__native-tls-0.2.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-native-tls-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-tls-0.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ipnet-2.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-pki-types-1.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-pemfile-2.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__reqwest-0.12.15
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-trait-0.1.88
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-http-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__glob-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-stream-0.1.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry_sdk-0.27.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__prost-derive-0.13.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__prost-0.13.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-stream-impl-0.3.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__async-stream-0.3.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-core-0.4.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchit-0.7.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-0.7.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-timeout-0.5.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__pin-project-internal-1.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-1.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__indexmap-1.9.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-0.4.13
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tonic-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-proto-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-otlp-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fastrand-2.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__linux-raw-sys-0.9.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustix-1.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tempfile-3.19.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__linux-raw-sys-0.4.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustix-0.38.44
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-mio-0.2.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossterm-0.28.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__coolor-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__strict-0.2.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__crokey-proc_macros-1.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crokey-1.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-channel-0.5.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-epoch-0.9.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-deque-0.8.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-queue-0.3.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aho-corasick-1.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-syntax-0.8.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-automata-0.4.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-1.11.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__lazy-regex-proc_macros-3.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lazy-regex-3.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minimad-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unicode-width-0.1.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__termimad-0.31.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-log-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-syntax-0.6.29
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-automata-0.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchers-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__overload-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nu-ansi-term-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sharded-slab-0.1.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thread_local-1.1.8
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-serde-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-subscriber-0.3.19
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-opentelemetry-0.28.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tracing-test-macro-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-test-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitflags-1.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__filetime-0.2.25
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__same-file-1.0.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__walkdir-2.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inotify-sys-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inotify-0.9.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mio-0.8.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__notify-5.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__semver-1.0.26
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sysinfo-0.28.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yaque-0.6.6
-Ldependency=bazel-out/aarch64-opt/bin/na/c
-Ldependency=bazel-out/aarch64-opt/bin/na/apps/ch
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64ct-1.7.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__blake2-0.10.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pass-hash-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__argon2-0.5.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-literal-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__env_logger-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quickcheck-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/a1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rayon-core-1.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rayon-1.10.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__keccak-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha3-0.10.8
-Ldependency=bazel-out/aarch64-opt/bin/a2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bi-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bio-0.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-conservative-0.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bh-0.14.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base58ck-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bech32-0.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitcoin-units-0.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex_lit-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__secp256k1-sys-0.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__secp256k1-0.29.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitcoin-0.32.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bcr-json-0.19.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minreq-2.13.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__jsonrpc-0.18.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bcr-0.19.0
-Ldependency=bazel-out/aarch64-opt/bin/nc-bitcoin-sync
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bs58-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__dynasm-3.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dt-3.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fixed-capacity-vec-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-0.4.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashx-0.3.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__visibility-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ex-0.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__data-encoding-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinyvec_macros-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinyvec-1.9.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-recursion-1.1.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__enum-as-inner-0.6.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hp-0.25.0-alpha.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tagptr-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__uuid-1.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__moka-0.12.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hostname-0.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__resolv-conf-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hr-0.25.0-alpha.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hmac-0.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hkdf-0.12.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__core2-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unsigned-varint-0.8.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multihash-0.19.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quick-protobuf-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-identity-0.2.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base-x-0.2.11
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__data-encoding-macro-internal-0.1.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__data-encoding-macro-0.1.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multibase-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multiaddr-0.18.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unsigned-varint-0.7.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multistream-select-0.13.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rw-stream-sink-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-core-0.43.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lru-0.12.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__libp2p-swarm-derive-0.35.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-swarm-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-allow-block-list-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-connection-limits-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__asynchronous-codec-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-bounded-0.2.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quick-protobuf-codec-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-identify-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crunchy-0.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__uint-0.10.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-kad-0.47.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-ping-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cbor4ii-0.3.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-request-response-0.28.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-dns-0.43.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memory-stats-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sysinfo-0.33.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-memory-connection-limits-0.4.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__paste-1.0.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-utils-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-core-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-route-0.17.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-sys-0.8.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-proto-0.11.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nix-0.26.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rtnetlink-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__if-watch-3.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__untrusted-0.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ring-0.17.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-webpki-0.103.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-0.23.26
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-rustls-0.26.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pem-3.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__untrusted-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__spin-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ring-0.16.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__powerfmt-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__deranged-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-conv-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__time-core-0.1.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_threads-0.1.7
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__time-macros-0.2.22
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__time-0.3.41
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yasna-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rcgen-0.11.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-webpki-0.101.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minimal-lexical-0.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nom-7.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rusticata-macros-4.1.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__asn1-rs-derive-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__asn1-rs-impl-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__asn1-rs-0.6.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-integer-0.1.46
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-bigint-0.4.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__der-parser-9.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__oid-registry-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__x509-parser-0.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-tls-0.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustc-hash-2.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-proto-0.11.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-udp-0.5.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-0.11.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-quic-0.12.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-0.55.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_bytes-0.11.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__void-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/nc-libp2p-io
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__termcolor-1.4.1
-Ldependency=bazel-out/aarch64-opt/bin/nc
-Ldependency=bazel-out/aarch64-opt/bin/nc-m
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anes-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cast-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-io-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__half-2.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-ll-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__itertools-0.14.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-plot-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__is-terminal-0.4.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__oorandom-11.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-backend-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-svg-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinytemplate-1.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_with_macros-3.12.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_with-3.12.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_repr-0.1.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bollard-stubs-1.47.1-rc.27.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__home-0.5.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-rustls-0.26.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-rustls-0.27.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-native-certs-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyperlocal-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bollard-0.18.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.21.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__docker_credential-1.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__etcetera-0.9.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__parse-display-derive-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parse-display-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__xattr-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-tar-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__testcontainers-0.23.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-open-sys-4.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-0.4.8
--sysroot=bazel-out/aarch64-opt/bin/external/rules_rust++rust+rust_linux_aarch64__aarch64-unknown-linux-gnu__nightly_tools/rust_toolchain
-Clinker-plugin-lto
-Copt-level=3
-Cdebug-assertions=off
-Coverflow-checks=off
-Ccodegen-units=1
-Cdebuginfo=1
-Cincremental=false
-Cembed-bitcode=yes
-Ctarget-cpu=native |
Here's the
Some of the more unusual bits is I was trying to isolate whether it was the Mac toolchain causing the weirdness by making it use an explicitly Bazel-provisioned LLVM toolchain. That ended up not mattering! The spooky-fast performance repros on aarch64 Linux inside Docker/OrbStack on my Mac. Note: I'm posting these snippets because I cannot put an MRE together from public/OSS components. It's a macro/end-to-end benchmark. I'll revisit the micro-benchmark suite in our project to see if there are any repros there in a bit.
|
Lightly censored Bazel run command for the benchmark:
The noise is my attempt at forcing it to use the The Cargo command is exactly what you'd expect: I get the same results regardless of whether I invoke the benchmark binary directly from |
@bitemyapp, I’ve been following this bug with interest. I’m not a committer, but I’m curious to know if you’ve been able to get to the root of the issue. |
@bitemyapp here you note that there is a perf difference between
If that's the case, the next thing to look into seems to be |
I'm going to try to assemble an MRE but it's going to take me a little time because I'm working inside of a proprietary monorepo. Let me describe what I'm seeing:
When building the same Rust codebase with both Cargo and Bazel on Apple Silicon (ARM64) and x86_64 Linux, the Bazel-built binary shows approximately 60% worse performance in benchmarks compared to the Cargo-built version, despite seemingly identical optimization settings.
Environment
Platform: macOS (ARM64) and Linux (x86_64)
Rust version: nightly-2025-02-07
rules_rust version: 0.60.0
Both builds use static linking
Both builds have identical environment variables
What I've Tried
linker-plugin-lto
only worked on Linux, barely helped (~1.5%)--subcommands
andRUSTC_LOG=rustc_codegen_ssa::back::lto=info
to confirm LTO was firing, it was on both platforms.rustc_flags
on the binary target, library targets, etc. a la https://github.com/bazelbuild/examples/tree/main/rust-examples/03-comp-optBut even with that it's quite a bit slower than the Cargo version.
Primary hypothesis: Dependency resolution method differences between Cargo and Bazel
My best guess is that the way the Bazel build links the
rust_library
targets defeats cross-crate inlining, monomorphization and optimization during LTO. My project's performance is extremely sensitive to inlining. Caveat: the most performance sensitive/critical libraries are bothextern
in Bazel and Cargo.Secondary hypotheses:
Reproduction Steps
The benchmark is
rust_binary
building abenches/
file that uses the Criterion benchmark harness intenrally.I think I need a way to force Bazel to use --extern-style dependency resolution rather than -L library paths, similar to how Cargo resolves dependencies in workspace mode, to get proper monomorphization and LTO behavior. I cannot use Bazel for this Rust project if this requires rewriting all the crate dependencies as one big source tree.
Frankly I'm a little baffled because I know major projects are using Bazel for their builds and I don't know how I'm running into a perf issue like this that I haven't been able to see mentioned anywhere on the web. And I looked very hard for answers. I've been working on this for the last few days.
Here's a sample
.bazelrc
I used in my testing:Linux:
macOS:
I am desperately hoping that I simply missed something obvious. My command for building the binary would typically be something like:
bazel build --config=release //crate:bench_name
and then I'd run the benchmark binary inbazel-bin/
with the--bench
argument.One difference between the binaries I noticed was that the Bazel generated binary was ~8.8 MiB vs. the ~8.3 MiB of the Cargo built binary. I first noticed the performance disparity with a very vanilla
bazel build -c opt
generated binary.toolchain setup from
MODULE.bazel
:Then the
rust_binary
for the benchmark:There are two particular dependencies which are exposed as
rust_library
s in the Bazel build which are very inlining sensitive, either one not getting inlined properly would explain the overall benchmark being slower without any obvious hot-spots. Comparing the params and rustc invocations from Cargo don't show any obvious differences. e.g.From Bazel, benchmark program params in bazel-out:
From Cargo:
Then buried in Cargo's deps:
I've been using
nm -C
andnm -gUC
with awk to generate symbol table counts and they're strongly similar. The main pattern I'm seeing is additional symbols getting included in the Bazel binary seemingly because dead-code elimination isn't firing for the library artifacts properly.Turning LTO off for both builds makes the Cargo and Bazel benchmarks about the same.
One difference I just noticed:
Bazel rustc for building the final benchmark binary:
Cargo, for the same:
Update: after some research, I think the
=path
argument toemit=link
is just setting the output path, shouldn't make a difference.It looks like the rlibs have both object code and bitcode in both Bazel and Cargo.
The
rust_library
definitions look like this:New update: enabling
build:release --@rules_rust//rust/settings:lto=thin
worked locally on Linux and macOS (still same perf issue tho) but barfed on the proc macro crates in CI/CD with a linker error that is indicative of "tried improperly to LTO a proc macro crate".Update: A Reddit user suggested a possibility, here's my reply:
Update, follow-up:
The text was updated successfully, but these errors were encountered: