Skip to content

Rust application much slower when built with rules_rust than with Cargo #3407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bitemyapp opened this issue Apr 14, 2025 · 20 comments
Open

Comments

@bitemyapp
Copy link

bitemyapp commented Apr 14, 2025

I'm going to try to assemble an MRE but it's going to take me a little time because I'm working inside of a proprietary monorepo. Let me describe what I'm seeing:

When building the same Rust codebase with both Cargo and Bazel on Apple Silicon (ARM64) and x86_64 Linux, the Bazel-built binary shows approximately 60% worse performance in benchmarks compared to the Cargo-built version, despite seemingly identical optimization settings.

Environment

Platform: macOS (ARM64) and Linux (x86_64)
Rust version: nightly-2025-02-07
rules_rust version: 0.60.0
Both builds use static linking
Both builds have identical environment variables

What I've Tried

  • Verified LTO is properly enabled in both builds (both thin and fat LTO, with and without LTO=manual)
  • linker-plugin-lto only worked on Linux, barely helped (~1.5%)
  • Confirmed optimizations are correctly set (opt-level=3, codegen-units=1)
  • Tried adding -Ctarget-cpu=native and ARM-specific target features on both platforms.
  • Verified the same code paths are being executed (same performance profile allocation percentages)
  • I used --subcommands and RUSTC_LOG=rustc_codegen_ssa::back::lto=info to confirm LTO was firing, it was on both platforms.
  • Manually setting rustc_flags on the binary target, library targets, etc. a la https://github.com/bazelbuild/examples/tree/main/rust-examples/03-comp-opt
  • The only thing that has significantly helped performance (-10%) has been to set this:
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cllvm-args=-import-instr-limit=5000
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cllvm-args=-inline-threshold=10000

But even with that it's quite a bit slower than the Cargo version.

Primary hypothesis: Dependency resolution method differences between Cargo and Bazel

My best guess is that the way the Bazel build links the rust_library targets defeats cross-crate inlining, monomorphization and optimization during LTO. My project's performance is extremely sensitive to inlining. Caveat: the most performance sensitive/critical libraries are both extern in Bazel and Cargo.

Secondary hypotheses:

  • Different LLVM optimization pass ordering between Cargo and Bazel builds
  • Dependency build order affecting optimization outcomes (I don't rate this highly)

Reproduction Steps

  • Build the same binary with both Cargo and Bazel using release configurations
  • Run benchmarks on both binaries (both outside of the sandbox)
  • Observe approximately 60% worse performance in the Bazel-built binary

The benchmark is rust_binary building a benches/ file that uses the Criterion benchmark harness intenrally.

I think I need a way to force Bazel to use --extern-style dependency resolution rather than -L library paths, similar to how Cargo resolves dependencies in workspace mode, to get proper monomorphization and LTO behavior. I cannot use Bazel for this Rust project if this requires rewriting all the crate dependencies as one big source tree.

Frankly I'm a little baffled because I know major projects are using Bazel for their builds and I don't know how I'm running into a perf issue like this that I haven't been able to see mentioned anywhere on the web. And I looked very hard for answers. I've been working on this for the last few days.

Here's a sample .bazelrc I used in my testing:

Linux:

build:release --action_env=RUSTC_LOG=rustc_codegen_ssa::back::lto=info
build:release --compilation_mode=opt
build:release --@rules_rust//rust/settings:lto=fat
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off

build:release --@rules_rust//rust/settings:extra_rustc_flag=-Clinker-plugin-lto
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cembed-bitcode=yes
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-fuse-ld=lld

macOS:

build:release --action_env=RUSTC_LOG=rustc_codegen_ssa::back::lto=info
build:release --compilation_mode=opt
build:release --@rules_rust//rust/settings:lto=fat
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off

I am desperately hoping that I simply missed something obvious. My command for building the binary would typically be something like:

bazel build --config=release //crate:bench_name and then I'd run the benchmark binary in bazel-bin/ with the --bench argument.

One difference between the binaries I noticed was that the Bazel generated binary was ~8.8 MiB vs. the ~8.3 MiB of the Cargo built binary. I first noticed the performance disparity with a very vanilla bazel build -c opt generated binary.

toolchain setup from MODULE.bazel:

bazel_dep(name = "rules_rust", version = "0.60.0")

rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")
rust.toolchain(
    edition = "2021",
    versions = ["nightly/2025-02-07"],
    dev_components = True,
)
use_repo(rust, "rust_toolchains")

register_toolchains("@rust_toolchains//:all")

[...]

crates.from_cargo(
    name = "crates",
    cargo_lockfile = "//:Cargo.lock",
    generate_binaries = True,
    manifests = [
       (Cargo.tomls of the workspace and constituent crates)
    ],
)

Then the rust_binary for the benchmark:

rust_binary(
    name = "my_benchmark",
    srcs = ["benches/my_benchmark.rs"],
    aliases = aliases(),
    edition = "2021",
    proc_macro_deps = all_crate_deps(
        proc_macro = True,
        proc_macro_dev = True,
    ),
    deps = [
        ":lib",
        (other in-workspace dependencies)
    ] + all_crate_deps(
        normal = True,
        normal_dev = True,
    ),
)

There are two particular dependencies which are exposed as rust_librarys in the Bazel build which are very inlining sensitive, either one not getting inlined properly would explain the overall benchmark being slower without any obvious hot-spots. Comparing the params and rustc invocations from Cargo don't show any obvious differences. e.g.

From Bazel, benchmark program params in bazel-out:

--extern=pkg-a=bazel-out/darwin_arm64-opt/bin/pkg-a/libpkg-a-262915972.rlib
-Ldependency=bazel-out/darwin_arm64-opt/bin/pkg-a
--extern=pkg-b=bazel-out/darwin_arm64-opt/bin/pkg-b/libpkg-b-2509656758.rlib
-Ldependency=bazel-out/darwin_arm64-opt/bin/pkg-b

From Cargo:

-L
dependency=/Users/callen/.cargo/cache/release/deps
--extern
pkg-a=/Users/callen/.cargo/cache/release/deps/libpkg-a-27b8eb42159f5c37.rlib
--extern
pkg-b=/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rlib

Then buried in Cargo's deps:

/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rmeta: (…source listing…)

/Users/callen/.cargo/cache/release/deps/libpkg-b-f26c14e53d574db5.rlib: (…source listing…)

/Users/callen/.cargo/cache/release/deps/pkg-b-f26c14e53d574db5.d: (…source listing…)

I've been using nm -C and nm -gUC with awk to generate symbol table counts and they're strongly similar. The main pattern I'm seeing is additional symbols getting included in the Bazel binary seemingly because dead-code elimination isn't firing for the library artifacts properly.

Turning LTO off for both builds makes the Cargo and Bazel benchmarks about the same.

One difference I just noticed:

Bazel rustc for building the final benchmark binary:

--emit=link=bazel-out/darwin_arm64-opt/bin/crate/benchmark
--emit=dep-info

Cargo, for the same:

--emit=dep-info,link

Update: after some research, I think the =path argument to emit=link is just setting the output path, shouldn't make a difference.

It looks like the rlibs have both object code and bitcode in both Bazel and Cargo.

The rust_library definitions look like this:

rust_library(
    name = "pkg-b",
    srcs = glob(["src/**/*.rs"]),
    crate_features = [
        "default",
    ],
    edition = "2021",
    proc_macro_deps = [
        "//pkg-b/macros",
    ] + all_crate_deps(proc_macro = True),
    deps = [
        "//pkg-b/vendored-crate-1",
        "//pkg-b/vendored-crate-2",
        "//pkg-b/vendored-crate-3",
        "//pkg-b/vendored-crate-4",
    ] + all_crate_deps(),
)
rust_library(
    name = "pkg-a",
    srcs = glob(["src/**/*.rs"]),
    aliases = aliases(),
    crate_features = [
        "default",
        "bazel_build",
    ],
    edition = "2021",
    proc_macro_deps = [
        "//pkg-b/macros",
    ] + all_crate_deps(proc_macro = True),
    deps = [
        "//pkg-c",
        "//pkg-b/vendored-crate-2",
        "//pkg-b",
    ] + all_crate_deps(),
)

New update: enabling build:release --@rules_rust//rust/settings:lto=thin worked locally on Linux and macOS (still same perf issue tho) but barfed on the proc macro crates in CI/CD with a linker error that is indicative of "tried improperly to LTO a proc macro crate".

Wondering if the CC and CXX compilers, and every other environment variable are the same between cargo and bazel. I think you may be compiling your sys dependencies differently.

Update: A Reddit user suggested a possibility, here's my reply:

I think you may be compiling your sys dependencies differently.

Could be, but all the perf impact is the pure Rust crates that live inside the Cargo workspace. There's very very little -sys in the dep tree and none of it is in the critical path. If any of it had been, it would've shown up when I profiled the optimized builds with debug symbols.

Update, follow-up:

    build:release --compilation_mode=opt
    build:release --@rules_rust//rust/settings:lto=thin
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=0
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cstrip=debuginfo
    build:release --@rules_rust//rust/settings:extra_rustc_flag=-Cforce-frame-pointers=no

This made the benchmark 3.5x as slow as the original benchmark with Cargo.

@lgarbarini
Copy link

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

@bitemyapp
Copy link
Author

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

I've been sorting, diffing, comparing the symbol tables with nm and the section sizes with bloaty but I'll give that a shot, thank you!

@lgarbarini
Copy link

Not a maintainer, but curious if you've used a tool like elf_diff to see what types of differences exist.

I've been sorting, diffing, comparing the symbol tables with nm and the section sizes with bloaty but I'll give that a shot, thank you!

Got it, I'd be curious to see some examples of short functions where there's a diff (something innocuous in terms of confidentiality).

@bitemyapp
Copy link
Author

@lgarbarini I haven't had a chance to try that tool yet but it does seem like the slowdown is evenly spread across two particular crates that really need to be fast and get inlined/LTO'd a lot. I don't know what elf_diff would show me that nm, samply, and bloaty haven't already, I've been comparing the symbol tables.

@lgarbarini
Copy link

@lgarbarini I haven't had a chance to try that tool yet but it does seem like the slowdown is evenly spread across two particular crates that really need to be fast and get inlined/LTO'd a lot. I don't know what elf_diff would show me that nm, samply, and bloaty haven't already, I've been comparing the symbol tables.

I think my message was unclear, completely agree that objdump, nm, samply, and bloaty have you covered, no need for elf_diff if you're doing it manually.

If the manual diff is exclusively inlined functions then I'll poke around some more, but if the diff contains other changes to other instructions outside of inlining, I would be curious to see a few excerpted sections from objdump.

@bitemyapp
Copy link
Author

bitemyapp commented Apr 15, 2025

Image

Oh what I wouldn't give for this tool to be written in Rust instead of Python.

@bitemyapp
Copy link
Author

While we're waiting for elf_diff to finish, here's how I've been comparing the symbols:

#!/usr/bin/env bash

export CARGO_BINARY=/home/callen/.cargo/cache/release/deps/benchmark-c0594d7ce94f306a
export BAZEL_BINARY=(...path to the bazel-bin/ benchmark binary)

nm -gUC $CARGO_BINARY | awk -F'::' '/ [Tt] / { print $2 }' | sort | uniq -c | sort -nr
nm -gUC $BAZEL_BINARY | awk -F'::' '/ [Tt] / { print $2 }' | sort | uniq -c | sort -nr

nm -C $CARGO_BINARY | grep pkg-b | wc -l
nm -C $BAZEL_BINARY | grep pkg-b | wc -l

nm -C $CARGO_BINARY | grep pkg-a | wc -l
nm -C $BAZEL_BINARY | grep pkg-a | wc -l

nm -gUC $CARGO_BINARY | grep pkg-b | wc -l
nm -gUC $BAZEL_BINARY | grep pkg-b | wc -l

nm -gUC $CARGO_BINARY | grep pkg-a | wc -l
nm -gUC $BAZEL_BINARY | grep pkg-a | wc -l

nm -gUC $CARGO_BINARY | grep sync | wc -l
nm -gUC $BAZEL_BINARY | grep sync | wc -l

# nm -gUC $CARGO_BINARY | grep sync | sort > /tmp/cargo_sync.txt
# nm -gUC $BAZEL_BINARY | grep sync | sort > /tmp/bazel_sync.txt
# diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

# nm -gUC $CARGO_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sort > /tmp/cargo_sync.txt
# nm -gUC $BAZEL_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sort > /tmp/bazel_sync.txt
# diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

nm -gUC $CARGO_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo_sync.txt
nm -gUC $BAZEL_BINARY | grep sync | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel_sync.txt
diff -u /tmp/cargo_sync.txt /tmp/bazel_sync.txt

nm -gUC $CARGO_BINARY | grep pkg-b | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | grep pkg-b | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

nm -gUC $CARGO_BINARY | grep pkg-a | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | grep pkg-a | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

nm -gUC $CARGO_BINARY | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/cargo.txt
nm -gUC $BAZEL_BINARY | awk '{$1=""; print $0}' | sed 's/^ *//' | sed -E 's/::h[0-9a-f]+$//' | sort > /tmp/bazel.txt
diff -u /tmp/cargo.txt /tmp/bazel.txt

The differences just aren't significant at least from what I can see from nm.

345
343
296
295
121
121
193
192
88
89

Is an example of the output for the middle commands in between the diffs and the uniq counts.

@bitemyapp
Copy link
Author

bitemyapp commented Apr 15, 2025

Someone in another thread raised it so I'll say it explicitly here: I have profiled this over and over. The slowdown is spread across the entire call graph of the 1 or 2 main crates that impact performance, there are no hot-spots, just deeply rooted parts of the call graph that get wider as a whole. I've been using samply to profile and it seems to be capturing everything but I'm running with cachegrind right now to try to get a more complete profile. If there was a hotspot I'd have ideas for what to try other than fiddling with .bazelrc over and over across the whole build.

@marvin-hansen
Copy link
Contributor

Wow, this is quite something. Just want to follow along to see what comes up in the investigation.
Also, please share a repro to give the community a chance to help in a meaningful way.

@marvin-hansen
Copy link
Contributor

marvin-hansen commented Apr 15, 2025

Also, have you verified that the resulting binaries are actually statically linked?

I have reported god knows how many issues related to static linking with rules_rust and ultimately given up.
However, I believe it was fixed a while ago. If you statically link with MUSL and your resulting binary is actually static, your culprit might be the MUSL default memory allocator especially under heavy concurrent workload. I saw a 10X drop in that case. Switching to MiMalloc recovered the 10x perf drop and then added some tiny boost on top of it. See this repo: https://github.com/marvin-hansen/mimalloc

@bitemyapp
Copy link
Author

@marvin-hansen There's only so much I can say about this which is related to why an MRE is so hard, but a few things:

  • I don't use musl because I'm paranoid
  • Our allocators are glibc malloc and mmap for an arena allocator, the critical path of the benchmark doesn't call malloc at all and actually aborts the process if it tries to. I generally use jemalloc for daemonized servers but malloc hasn't been a designated for this project yet. I try to wait until I have metrics/profiles before I change things from the platform defaults.

My main update at this point is that I'm no longer able to reproduce the original Cargo baseline benchmark performance on Linux…with Cargo. Even if I backtrack to older commits. I can still repro it on Mac + Cargo.

It's a macro-benchmark that involves a very thick stack of technology, I can't really make an MRE that meaningfully captures this. I will scrutinize my micro-benchmarks and see if any of them are evincing the same slow-down factor. If such a thing exists, then I will extract an MRE, but there's always a chance that whatever fixes the micro-benchmark doesn't fix anything for the big boy.

@bitemyapp
Copy link
Author

@lgarbarini I looked at the elf_diff results this morning, I saw some false deleted/added where it was the same symbol with the same asm read-out getting flagged for unknown reasons. Nothing in particular stood out but I haven't had time to scrutinize it carefully. I'm going to focus on an MRE and shrinking the repro because there's just too much stuff happening in this benchmark and I need to isolate variables.

@bitemyapp
Copy link
Author

bitemyapp commented Apr 26, 2025

Yeah I'm still having this perf issue with the Bazel-built version of my program. I managed to get some unrelated issues fixed and got back to the original baseline performance for the Cargo and Bazel versions, same situation as before. As near as I can tell it's like the usual LTO and LLVM optimizations aren't happening. An example is that we have an empty inline asm! hack as a branch hint, I'm pretty sure it's not getting optimized out in the Bazel version but I haven't confirmed yet. We rely on LLVM to do that. Another thing I'll note here is that the problem libm had which @tgross35 was able to fix with #[inline] on a generic function isn't likely applicable here. We have non-generic functions with #[inline] or more commonly #[inline(always)] on them.

libm thing: rust-lang/libm#545

Update: I replaced the asm! hack with core::hint::cold_path and both builds got faster by the same coefficient.

Update2: I can only repro this on macOS, not Linux. This is increasingly looking like something weird with macOS or ld64, but I'm not sure.

@bitemyapp
Copy link
Author

bitemyapp commented Apr 27, 2025

Nope, it's repro'ing with Cargo inside of Docker + OrbStack + Ubuntu 24.10 + aarch64 on my M3 Pro Mac.

Suspiciously faster execution:

  • Cargo + Apple Silicon macOS (~6-7 seconds benchmark time)
  • Cargo + Apple Silicon Ubuntu 24.10 + Docker/OrbStack on M3 Pro Mac (~6-7s)
  • Cargo + Google Axion (aarch64) Ubuntu 24.10 (~9.8s)

Slower execution:

  • Manual rustc build + run (verbatim copy of the Cargo build in a script), Apple Silicon, macOS (~12-13s)
  • cargo bench --no-run then running the binary manually (~12-13s)
  • cargo run --bin my_benchmark (made a [[bin]] target for it in Cargo.toml) (~12-13s)
  • Bazel + Google Axion (aarch64) Ubuntu 24.10 (19.0s)
  • Bazel + Apple Silicon macOS (~12-13s)
  • Bazel + Apple Silicon Ubuntu 24.10 + Docker/OrbStack on M3 Pro Mac (~12-13s)
  • Cargo + x64 Linux (~12-13s)
  • Bazel + x64 Linux (~12-13s)

My local test bench for Linux is a 9800X3D + RTX 5090 desktop that I built recently, the benchmark is single-threaded and the single-threaded performance of the M3 Pro and 9800X3D are generally very close which is convenient for my purposes.

The commonality here so far seems to be cargo + aarch64 rather than macOS/Mac toolchain.

Update: there seems to be something specifically about letting cargo bench run the binary. Running the binary manually whether cargo bench, a verbatim rustc facsimile of the cargo bench -vv rustc command, or Bazel built it yields the slower result. I dumped the ENV from the benchmark program and there weren't any interesting differences. Only thing that flagged up was DYLD_FALLBACK_LIBRARY_PATH and I tried setting the script harness' DYLD_FALLBACK_LIBRARY_PATH for my manual rustc facsimile of cargo bench to the same value and it wasn't any faster. In addition, this benchmark is 100% Rust so far as I'm aware with zero C/dylib/*-sys code involved. I'm double-checking on that to increase my certainty.

@bitemyapp
Copy link
Author

Optimization flag perf impact:

Docker, OrbStack, M3 Mac, Bazel:

(these are sequential performance steps/changes)

  • lto=thin -> lto=off: none
  • codegen-units=1 -> (opt default, 16 I think): ~4.5% slower
  • opt-level=3 -> opt-level=2: 1.2% slower

Docker, OrbStack, M3 Mac, Cargo:

(sequential again)

  • lto=thin -> lto=off: 2.5% slower
  • codegen-units=1 -> (release default, 16?): 14.4% slower
  • opt-level=3 -> opt-level=2: oddly, it actually improves by ~1.2% on opt-level=2 unlike all the other instances.

lto=fat has only made a small difference unless I start messing with the LLVM inlining limits.

When I disabled codegen-units=1 for Cargo I disabled it for both the direct release profile but also for the package dependencies under profile.release.package."*". Ditto opt-level.

@bitemyapp
Copy link
Author

After writing a sed script for redacting some sensitive crate names, here are the full final build commands for the benchmark binary from Cargo and Bazel inside the Dockerfile:

cargo

CARGO=/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/bin/cargo
CARGO_BIN_EXE_app-nm=/app/target/release/app-nm
CARGO_CRATE_NAME=my_benchmark
CARGO_MANIFEST_DIR=/app/app-nm
CARGO_MANIFEST_PATH=/app/app-nm/Cargo.toml
CARGO_PKG_AUTHORS=''
CARGO_PKG_DESCRIPTION=''
CARGO_PKG_HOMEPAGE=''
CARGO_PKG_LICENSE=''
CARGO_PKG_LICENSE_FILE=''
CARGO_PKG_NAME=app-nm
CARGO_PKG_README=README.md
CARGO_PKG_REPOSITORY=''
CARGO_PKG_RUST_VERSION=''
CARGO_PKG_VERSION=0.1.0
CARGO_PKG_VERSION_MAJOR=0
CARGO_PKG_VERSION_MINOR=1
CARGO_PKG_VERSION_PATCH=0
CARGO_PKG_VERSION_PRE=''
CARGO_PRIMARY_PACKAGE=1
CARGO_SBOM_PATH=''
CARGO_TARGET_TMPDIR=/app/target/tmp
LD_LIBRARY_PATH='/app/target/release/deps:/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/lib'
/root/.rustup/toolchains/nightly-2025-04-26-aarch64-unknown-linux-gnu/bin/rustc
--crate-name
my_benchmark
--edition=2021
app-nm/benches/my_benchmark.rs
--error-format=json
--json=diagnostic-rendered-ansi,artifacts,future-incompat
--diagnostic-width=106
--emit=dep-info,link
-C
opt-level=3
-C
lto=thin
-C
codegen-units=1
-C
debuginfo=1
--cfg
test
--check-cfg
'cfg(docsrs,test)'
--check-cfg
'cfg(feature,
values())'
-C
metadata=65b29445035900e0
-C
extra-filename=-5a438e8f8ec6ae3d
--out-dir
/app/target/release/deps
-L
dependency=/app/target/release/deps
--extern
anyhow=/app/target/release/deps/libanyhow-b1edd40a0e594b17.rlib
--extern
bs58=/app/target/release/deps/libbs58-fae0e5699db31778.rlib
--extern
nm-ch=/app/target/release/deps/libnm-ch-68f6ac10eeca7a26.rlib
--extern
chrono=/app/target/release/deps/libchrono-6c2368740715bfe1.rlib
--extern
clap=/app/target/release/deps/libclap-9acdcde543eb8016.rlib
--extern
criterion=/app/target/release/deps/libcriterion-f607b0ecd2564939.rlib
--extern
nm_c=/app/target/release/deps/libnm-c-0bb82d835be9c8cf.rlib
--extern
either=/app/target/release/deps/libeither-f430a192a1c2d6c9.rlib
--extern
getrandom=/app/target/release/deps/libgetrandom-ee380435cb066fcd.rlib
--extern
gnort=/app/target/release/deps/libgnort-330afad64b885c9e.rlib
--extern
ibig=/app/target/release/deps/libibig-eb5fc25037ace091.rlib
--extern
artifacts=/app/target/release/deps/libartifacts-df5162a314165126.rlib
--extern
libc=/app/target/release/deps/liblibc-cf0d128ff4389ed5.rlib
--extern
l2=/app/target/release/deps/libl2-f79f90140ac6f04a.rlib
--extern
nmnc=/app/target/release/deps/libnmnc-47a721e54fbfdca0.rlib
--extern
nmnc_l2=/app/target/release/deps/libnmnc_l2-cc7422f92be1ebd4.rlib
--extern
nmnc_m=/app/target/release/deps/libnmnc_m-b09243d00dce314e.rlib
--extern
num_cpus=/app/target/release/deps/libnum_cpus-1411bfdf63d9102e.rlib
--extern
perf_event=/app/target/release/deps/libperf_event-773f2ee67ac5840f.rlib
--extern
pin_project_lite=/app/target/release/deps/libpin_project_lite-d72327f10b8ee167.rlib
--extern
rand=/app/target/release/deps/librand-5057525688919945.rlib
--extern
reqwest=/app/target/release/deps/libreqwest-f599cef6ebc0021c.rlib
--extern
serde=/app/target/release/deps/libserde-68bb54fbdf39400f.rlib
--extern
serde_json=/app/target/release/deps/libserde_json-ccacf4f8583e1327.rlib
--extern
nms=/app/target/release/deps/libnms-56e9cf15c14b887b.rlib
--extern
nms_macros=/app/target/release/deps/libnms_macros-bc968a627bfa56a4.so
--extern
tempfile=/app/target/release/deps/libtempfile-c8c38e0b986b3748.rlib
--extern
testcontainers=/app/target/release/deps/libtestcontainers-d4eb9115a94b91b1.rlib
--extern
thiserror=/app/target/release/deps/libthiserror-df954635be403dad.rlib
--extern
tokio=/app/target/release/deps/libtokio-9dc8c59b1c711c7d.rlib
--extern
tokio_util=/app/target/release/deps/libtokio_util-7fd2c8732417e306.rlib
--extern
tracing=/app/target/release/deps/libtracing-c0e09c928a721c66.rlib
--extern
tracing_subscriber=/app/target/release/deps/libtracing_subscriber-967fe0e0afece883.rlib
--extern
utilities=/app/target/release/deps/libutilities-312805f8dc6cf16a.rlib
--extern
a2=/app/target/release/deps/liba2-d95cb3346f5f17ca.rlib
-L
native=/app/target/release/build/blake3-319ac08c6f4e13fb/out
-L
native=/app/target/release/build/ring-bb137da75991c7b4/out
-L
native=/app/target/release/build/ring-6e5ba74e6f60ab19/out
-L
native=/app/target/release/build/secp256k1-sys-4b18d0f5e952cbe1/out

bazel

nc-m/benches/my_benchmark.rs
--crate-name=my_benchmark
--crate-type=bin
--error-format=human
--out-dir=bazel-out/aarch64-opt/bin/nc-m
--codegen=opt-level=3
--codegen=debuginfo=0
--codegen=strip=debuginfo
--remap-path-prefix=${pwd}=
--emit=link=bazel-out/aarch64-opt/bin/nc-m/my_benchmark
--emit=dep-info
--color=always
--target=aarch64-unknown-linux-gnu
-L
bazel-out/aarch64-opt/bin/external/rules_rust++rust+rust_linux_aarch64__aarch64-unknown-linux-gnu__nightly_tools/rust_toolchain/lib/rustlib/aarch64-unknown-linux-gnu/lib
--edition=2021
-Clto=thin
--codegen=linker=external/toolchains_llvm++llvm+llvm_toolchain/bin/cc_wrapper.sh
--codegen=link-arg=--target=aarch64-unknown-linux-gnu
--codegen=link-arg=-no-canonical-prefixes
--codegen=link-arg=-lm
--codegen=link-arg=-fuse-ld=lld
--codegen=link-arg=-Wl,--build-id=md5
--codegen=link-arg=-Wl,--hash-style=gnu
--codegen=link-arg=-Wl,-z,relro,-z,now
--codegen=link-arg=-l:libc++.a
--codegen=link-arg=-l:libc++abi.a
--codegen=link-arg=-l:libunwind.a
--codegen=link-arg=-lpthread
--codegen=link-arg=-ldl
--codegen=link-arg=-rtlib=compiler-rt
--codegen=link-arg=-Wl,--gc-sections
--extern=nc_m=bazel-out/aarch64-opt/bin/nc-m/libnc_m-3221654267.rlib
--extern=ks=bazel-out/aarch64-opt/bin/ks/libks-2401207873.rlib
--extern=c=bazel-out/aarch64-opt/bin/na/c/libc-401854362.rlib
--extern=nc=bazel-out/aarch64-opt/bin/nc/libnc-2793115565.rlib
--extern=s=bazel-out/aarch64-opt/bin/s/rust/s/libs-262915972.rlib
--extern=a2=bazel-out/aarch64-opt/bin/a2/liba2-894933967.rlib
--extern=bs58=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bs58-0.5.1/libbs58-1951919875.rlib
--extern=clap=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap-4.5.37/libclap-186282177.rlib
--extern=either=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__either-1.15.0/libeither-1753065134.rlib
--extern=getrandom=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.2.15/libgetrandom-249734380.rlib
--extern=gnort=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__gnort-0.1.2/libgnort-1317023791.rlib
--extern=libc=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libc-0.2.172/liblibc-2400173565.rlib
--extern=libp2p=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-0.55.0/liblibp2p-1954658880.rlib
--extern=num_cpus=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_cpus-1.16.0/libnum_cpus-1232792777.rlib
--extern=pin_project_lite=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-lite-0.2.16/libpin_project_lite-2218591079.rlib
--extern=rand=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.8.5/librand-707188286.rlib
--extern=thiserror=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-2.0.12/libthiserror-2502407963.rlib
--extern=tokio=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-1.44.2/libtokio-2604446065.rlib
--extern=tokio_util=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-util-0.7.14/libtokio_util-50890834.rlib
--extern=tracing=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-0.1.41/libtracing-2194975523.rlib
--extern=tracing_subscriber=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-subscriber-0.3.19/libtracing_subscriber-324201358.rlib
--extern=anyhow=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anyhow-1.0.98/libanyhow-787701661.rlib
--extern=chrono=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__chrono-0.4.40/libchrono-937817766.rlib
--extern=criterion=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-0.5.1/libcriterion-161828024.rlib
--extern=reqwest=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__reqwest-0.12.15/libreqwest-737296596.rlib
--extern=serde=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde-1.0.219/libserde-498884776.rlib
--extern=serde_json=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_json-1.0.140/libserde_json-927511813.rlib
--extern=tempfile=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tempfile-3.19.1/libtempfile-1663650551.rlib
--extern=testcontainers=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__testcontainers-0.23.3/libtestcontainers-1979583287.rlib
--extern=perf_event=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-0.4.8/libperf_event-898811581.rlib
-Ldependency=bazel-out/aarch64-opt/bin/ks
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/assert_no_alloc
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cfg-if-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-traits-0.2.19
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerocopy-0.8.24
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ppv-lite86-0.2.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libc-0.2.172
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.2.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_core-0.6.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_chacha-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.8.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__static_assertions-1.1.0
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/ibig
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/murmur3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__typenum-1.18.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__generic-array-0.14.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crypto-common-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inout-0.1.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cipher-0.4.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cpufeatures-0.2.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aes-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aead-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dbl-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__block-buffer-0.10.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__subtle-2.6.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__digest-0.10.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cmac-0.7.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ctr-0.9.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zeroize_derive-1.4.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zeroize-1.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aes-siv-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__curve25519-dalek-4.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signature-2.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ed25519-2.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha2-0.10.8
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ed25519-dalek-2.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha1-0.10.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__x25519-dalek-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/s_crypto
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__funty-2.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__radium-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tap-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__wyz-0.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitvec-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__either-1.15.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__intmap-3.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__json-0.12.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lazy_static-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memmap2-0.9.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-registry-1.4.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-0.3.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__slotmap-1.0.7
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__thiserror-impl-2.0.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-2.0.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__log-0.4.27
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-lite-0.2.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__once_cell-1.21.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-core-0.1.33
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tracing-attributes-0.1.28
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-0.1.41
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/s/rust/s_macros
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__num-derive-0.4.2
-Ldependency=bazel-out/aarch64-opt/bin/s/rust/s
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anyhow-1.0.98
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_derive-1.0.219
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde-1.0.219
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bytes-1.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-core-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fnv-1.0.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__itoa-1.0.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-1.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-body-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__http-body-util-0.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mime-0.3.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sync_wrapper-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-layer-0.3.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-service-0.3.3
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__rustversion-1.0.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-core-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__percent-encoding-2.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__form_urlencoded-1.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-sink-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-channel-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-io-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-task-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memchr-2.7.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-utils-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__slab-0.4.9
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__futures-macro-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-util-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__atomic-waker-1.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__equivalent-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__allocator-api2-0.2.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foldhash-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.15.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__indexmap-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mio-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__scopeguard-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lock_api-0.4.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__smallvec-1.15.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parking_lot_core-0.9.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parking_lot-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__socket2-0.5.9
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tokio-macros-2.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-1.44.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-util-0.7.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__h2-0.4.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__httparse-1.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__httpdate-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__try-lock-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__want-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-1.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-util-0.1.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchit-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ryu-1.0.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_json-1.0.140
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_path_to_error-0.1.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_urlencoded-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-0.8.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unty-0.0.4
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__bincode_derive-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bincode-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__arrayref-0.3.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__arrayvec-0.7.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__constant_time_eq-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__blake3-1.5.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__byteorder-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__iana-time-zone-0.1.63
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__chrono-0.4.40
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-1.0.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf8parse-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-parse-0.2.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstyle-query-1.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__colorchoice-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__is_terminal_polyfill-1.70.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anstream-0.6.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap_lex-0.7.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__strsim-0.11.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap_builder-4.5.37
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__clap_derive-4.5.32
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__clap-4.5.37
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__option-ext-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dirs-sys-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dirs-6.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_cpus-1.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-executor-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-0.3.31
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-utils-0.8.21
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.14.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dashmap-6.1.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__derive_more-impl-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__derive_more-2.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__getrandom-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_core-0.9.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand_chacha-0.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rand-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__retry-2.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dogstatsd-0.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-timer-3.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__no-std-compat-0.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nonzero_ext-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__portable-atomic-1.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quanta-0.12.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__spinning_top-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__web-time-1.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__governor-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__maplit-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__gnort-0.1.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__thiserror-impl-1.0.69
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thiserror-1.0.69
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-0.27.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.22.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__stable_deref_trait-1.2.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zerofrom-derive-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerofrom-0.1.6
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__yoke-derive-0.7.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yoke-0.7.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__zerovec-derive-0.10.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__zerovec-0.10.4
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__displaydoc-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_collections-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_normalizer_data-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__litemap-0.7.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinystr-0.7.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__writeable-0.5.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid_transform_data-1.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__icu_provider_macros-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_provider-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_locid_transform-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_properties_data-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_properties-1.5.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf16_iter-1.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__utf8_iter-1.0.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__write16-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__icu_normalizer-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__idna_adapter-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__idna-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__url-2.5.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__encoding_rs-0.8.35
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitflags-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foreign-types-shared-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__foreign-types-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-sys-0.9.107
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__openssl-macros-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-0.10.72
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__openssl-probe-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__native-tls-0.2.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-native-tls-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-tls-0.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ipnet-2.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-pki-types-1.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-pemfile-2.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__reqwest-0.12.15
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-trait-0.1.88
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-http-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__glob-0.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-stream-0.1.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry_sdk-0.27.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__prost-derive-0.13.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__prost-0.13.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-stream-impl-0.3.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__async-stream-0.3.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-core-0.4.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchit-0.7.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__axum-0.7.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-timeout-0.5.2
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__pin-project-internal-1.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pin-project-1.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashbrown-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__indexmap-1.9.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tower-0.4.13
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tonic-0.12.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-proto-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__opentelemetry-otlp-0.27.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fastrand-2.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__linux-raw-sys-0.9.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustix-1.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tempfile-3.19.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__linux-raw-sys-0.4.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustix-0.38.44
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__signal-hook-mio-0.2.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossterm-0.28.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__coolor-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__strict-0.2.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__crokey-proc_macros-1.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crokey-1.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-channel-0.5.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-epoch-0.9.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-deque-0.8.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-queue-0.3.12
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crossbeam-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__aho-corasick-1.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-syntax-0.8.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-automata-0.4.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-1.11.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__lazy-regex-proc_macros-3.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lazy-regex-3.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minimad-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unicode-width-0.1.14
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__termimad-0.31.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-log-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-syntax-0.6.29
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__regex-automata-0.1.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__matchers-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__overload-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nu-ansi-term-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sharded-slab-0.1.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__thread_local-1.1.8
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-serde-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-subscriber-0.3.19
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-opentelemetry-0.28.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__tracing-test-macro-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tracing-test-0.2.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitflags-1.3.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__filetime-0.2.25
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__same-file-1.0.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__walkdir-2.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inotify-sys-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__inotify-0.9.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__mio-0.8.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__notify-5.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__semver-1.0.26
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sysinfo-0.28.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yaque-0.6.6
-Ldependency=bazel-out/aarch64-opt/bin/na/c
-Ldependency=bazel-out/aarch64-opt/bin/na/apps/ch
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64ct-1.7.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__blake2-0.10.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pass-hash-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__argon2-0.5.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-literal-1.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__env_logger-0.8.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quickcheck-1.0.3
-Ldependency=bazel-out/aarch64-opt/bin/a1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rayon-core-1.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rayon-1.10.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__keccak-0.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sha3-0.10.8
-Ldependency=bazel-out/aarch64-opt/bin/a2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bi-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bio-0.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-conservative-0.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bh-0.14.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base58ck-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bech32-0.11.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitcoin-units-0.1.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex_lit-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__secp256k1-sys-0.10.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__secp256k1-0.29.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bitcoin-0.32.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bcr-json-0.19.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minreq-2.13.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__jsonrpc-0.18.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bcr-0.19.0
-Ldependency=bazel-out/aarch64-opt/bin/nc-bitcoin-sync
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bs58-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__dynasm-3.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__dt-3.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__fixed-capacity-vec-1.0.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hex-0.4.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hashx-0.3.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__visibility-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ex-0.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__data-encoding-2.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinyvec_macros-0.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinyvec-1.9.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__async-recursion-1.1.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__enum-as-inner-0.6.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hp-0.25.0-alpha.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tagptr-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__uuid-1.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__moka-0.12.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hostname-0.4.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__resolv-conf-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hr-0.25.0-alpha.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hmac-0.12.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hkdf-0.12.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__core2-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unsigned-varint-0.8.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multihash-0.19.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quick-protobuf-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-identity-0.2.10
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base-x-0.2.11
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__data-encoding-macro-internal-0.1.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__data-encoding-macro-0.1.18
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multibase-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multiaddr-0.18.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__unsigned-varint-0.7.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__multistream-select-0.13.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rw-stream-sink-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-core-0.43.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__lru-0.12.5
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__libp2p-swarm-derive-0.35.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-swarm-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-allow-block-list-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-connection-limits-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__asynchronous-codec-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-bounded-0.2.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quick-protobuf-codec-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-identify-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__crunchy-0.2.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__uint-0.10.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-kad-0.47.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-ping-0.46.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cbor4ii-0.3.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-request-response-0.28.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-dns-0.43.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__memory-stats-1.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__sysinfo-0.33.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-memory-connection-limits-0.4.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__paste-1.0.15
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-utils-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-core-0.7.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-packet-route-0.17.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-sys-0.8.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__netlink-proto-0.11.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nix-0.26.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rtnetlink-0.13.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__if-watch-3.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__untrusted-0.9.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ring-0.17.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-webpki-0.103.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-0.23.26
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__futures-rustls-0.26.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__pem-3.0.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__untrusted-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__spin-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ring-0.16.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__powerfmt-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__deranged-0.4.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-conv-0.1.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__time-core-0.1.4
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num_threads-0.1.7
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__time-macros-0.2.22
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__time-0.3.41
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__yasna-0.5.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rcgen-0.11.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-webpki-0.101.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__minimal-lexical-0.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__nom-7.1.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rusticata-macros-4.1.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__asn1-rs-derive-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__asn1-rs-impl-0.2.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__asn1-rs-0.6.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-integer-0.1.46
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__num-bigint-0.4.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__der-parser-9.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__oid-registry-0.7.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__x509-parser-0.16.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-tls-0.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustc-hash-2.1.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-proto-0.11.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-udp-0.5.11
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__quinn-0.11.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-quic-0.12.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__libp2p-0.55.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_bytes-0.11.17
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__void-1.0.2
-Ldependency=bazel-out/aarch64-opt/bin/nc-libp2p-io
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__termcolor-1.4.1
-Ldependency=bazel-out/aarch64-opt/bin/nc
-Ldependency=bazel-out/aarch64-opt/bin/nc-m
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__anes-0.1.6
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__cast-0.3.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-io-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__half-2.6.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-ll-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__ciborium-0.2.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__itertools-0.14.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-plot-0.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__is-terminal-0.4.16
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__oorandom-11.1.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-backend-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-svg-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__plotters-0.3.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tinytemplate-1.2.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__criterion-0.5.1
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_with_macros-3.12.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__serde_with-3.12.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__serde_repr-0.1.20
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bollard-stubs-1.47.1-rc.27.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__home-0.5.9
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-rustls-0.26.2
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyper-rustls-0.27.5
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__rustls-native-certs-0.8.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__hyperlocal-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__bollard-0.18.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__base64-0.21.7
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__docker_credential-1.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__etcetera-0.9.0
-Ldependency=bazel-out/aarch64-opt-exec-ST-d57f47055a04/bin/external/rules_rust++crate+crates__parse-display-derive-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__parse-display-0.9.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__xattr-1.5.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__tokio-tar-0.3.1
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__testcontainers-0.23.3
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-open-sys-4.0.0
-Ldependency=bazel-out/aarch64-opt/bin/external/rules_rust++crate+crates__perf-event-0.4.8
--sysroot=bazel-out/aarch64-opt/bin/external/rules_rust++rust+rust_linux_aarch64__aarch64-unknown-linux-gnu__nightly_tools/rust_toolchain
-Clinker-plugin-lto
-Copt-level=3
-Cdebug-assertions=off
-Coverflow-checks=off
-Ccodegen-units=1
-Cdebuginfo=1
-Cincremental=false
-Cembed-bitcode=yes
-Ctarget-cpu=native

@bitemyapp
Copy link
Author

bitemyapp commented Apr 27, 2025

Here's the .bazelrc, this isn't super important honestly, I've tried hundreds of variations of these flags and have never ever gotten the benchmark numbers that Cargo gets on aarch64 with Bazel. My commented-out .bazelrc flags-graveyard was over 150 lines long until I cleaned it up recently.

build --@rules_rust//rust/toolchain/channel=nightly
build --compilation_mode=opt
build --@rules_rust//rust/settings:lto=thin

build --@rules_rust//rust/settings:extra_rustc_flag=-Copt-level=3
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebug-assertions=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Coverflow-checks=off
build --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=1
build --@rules_rust//rust/settings:extra_rustc_flag=-Cincremental=false
build --repo_env=CC=@llvm_toolchain//:bin/clang
build --repo_env=CXX=@llvm_toolchain//:bin/clang++
build --repo_env=LD=@llvm_toolchain//:bin/ld.lld
build:linux --@rules_rust//rust/settings:extra_rustc_flag=-Clinker-plugin-lto
build:macos --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-fuse-ld=lld
build:macos --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-Wl,-lto_library,/usr/lib/libLTO.dylib
build:macos --@rules_rust//rust/settings:extra_rustc_flag=-Clink-arg=-Wl,-cache_path_lto,/tmp/lto-cachebuild
build --@rules_rust//rust/settings:toolchain_generated_sysroot=True
build --@rules_rust//rust/settings:extra_rustc_flag=-Cembed-bitcode=yes
build --@rules_rust//rust/settings:extra_rustc_flag=-Ctarget-cpu=native

Some of the more unusual bits is I was trying to isolate whether it was the Mac toolchain causing the weirdness by making it use an explicitly Bazel-provisioned LLVM toolchain. That ended up not mattering! The spooky-fast performance repros on aarch64 Linux inside Docker/OrbStack on my Mac.

Note: I'm posting these snippets because I cannot put an MRE together from public/OSS components. It's a macro/end-to-end benchmark. I'll revisit the micro-benchmark suite in our project to see if there are any repros there in a bit.

MODULE.bazel is boring, rules_rust 0.60.0.toolchains_llvm 1.4.0, llvm_version = 20.1.0. rustc version is nightly/2025-04-26. All these issues repro with 1.86.0 stable and I'm using nightly for unrelated perf reasons.

@bitemyapp
Copy link
Author

bitemyapp commented Apr 27, 2025

Dockerfile I ran the Bazel and Cargo builds inside of on my M3 Pro Mac:

FROM --platform=linux/arm64 ubuntu:24.10
RUN apt update && apt upgrade -y && apt install -y libgcc-s1 curl build-essential git libxml2-dev pkg-config libssl-dev nodejs
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN npm install -g @bazel/bazelisk

Lightly censored Bazel run command for the benchmark:

RUST_BACKTRACE=full bazel run //nc-m:my_benchmark --@rules_rust//rust/settings:experimental_use_cc_common_link=false --extra_toolchains=@llvm_toolchain//:cc-toolchain-aarch64-linux --subcommands

The noise is my attempt at forcing it to use the MODULE.bazel provisioned LLVM toolchain (no idea if that's actually working or not suspect not given common link is false) and dumping the rustc commands.

The Cargo command is exactly what you'd expect: cargo bench -p nm-c --bench my_benchmark

I get the same results regardless of whether I invoke the benchmark binary directly from bazel-bin in my native environment or I use bazel run and let it use the default sandbox.

@pdeva
Copy link

pdeva commented May 15, 2025

@bitemyapp, I’ve been following this bug with interest. I’m not a committer, but I’m curious to know if you’ve been able to get to the root of the issue.

@scentini
Copy link
Collaborator

@bitemyapp here you note that there is a perf difference between cargo bench & cargo-based builds ran by invoking the binary directly. Apart from this entry:

Bazel + Google Axion (aarch64) Ubuntu 24.10 (19.0s)
the Bazel based builds seem on par with the cargo based ones. Could you confirm that you ran the cargo non cargo bench workflow in the equivalent environment, e.g:

  • cargo bench --no-run then running the binary manually, on Google Axion (aarch64) Ubuntu 24.10

If that's the case, the next thing to look into seems to be -Ldependency, as discussed on Zulip. If I can create a rules_rust patch that will result in a single -Ldependency on the Bazel side, would you be able to use the patch to re-run the benchmarks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants