Benchmarks of private code not running on macOS

[Fraser999]

I created a small repo to demonstrate the issue.

In the root, if I run cargo bench on Ubuntu, I see the expected benchmarking output, but on macOS, I don't.

[dvdplm]

This is what I see too. The "register anywhere" feature seems to be broken on macOS (I'm on Sequoia, 15.0 (24A335)).

[dvdplm]

@Fraser999 Did you by any chance dig deeper into this yet? Do you have a hunch about where the problem lies and/or which macOS version broke it?

[Fraser999]

Did you by any chance dig deeper into this yet?

No, I'm afraid I've not had time.

Do you have a hunch about where the problem lies and/or which macOS version broke it?

Also no. For reference, I'm on Sonoma (14.6.1). Potentially it has never worked on macOS, since the example here works for me and that's what's used in CI to test this functionality.

[dvdplm]

Yeah the example works for me too, always did. Never figured out how to do it outside the divan crate though.

It's such a killer feature, sad it's incomplete.

[jdidion]

The best solution I've found so far is to a) create a feature flag that you can use to conditionally compile benchmark modules, b) put the divan::main() call inside a main function in a benchmarks sub-module annotated with #[cfg(feature = "bench")], and c) call the crate::benchmarks::main function from the main function in the benches folder.

Note that you should only call divan::main() once. Calling it within the crate appears to register all functions anywhere in the crate, but calling it from code in the benches folder does not. I've adopted the practice of putting it in a sub-module of the top-level module (lib.rs or main.rs) in my crate.

So your cargo.toml looks like this:

[package]
name = "foo"
...

[dependencies]
divan = { version = "0.1.14", optional = true }

[features]
bench = ["dep:divan"]

[[bench]]
name = "benchmark"
harness = false

You have lib.rs:

mod bar;
mod baz;

#[cfg(feature = "bench")]
pub mod benchmarks {
    pub fn main() {
        divan::main();
    }

    // This is a good place to put any common code that is only
    // used in benchmarking
    pub fn my_bench_util() { ... }
}

You have bar.rs (baz.rs is similar):

fn my_private_fn() {
   ...
}

#[cfg(feature = "bench")]
mod benchmarks {
    use crate::benchmarks::my_bench_util;

    fn bench_my_private_fn() {
        my_bench_util();
        divan::black_box(super::my_private_fn());
    } 
}

And you have a benches/benchmark.rs file with the following:

fn main() {
    foo::benchmarks::main();
}

To run benchmarks: cargo bench --features bench

This isn't as nice as having the divan::main() call in benches but it's not too much worse - until #[cfg(bench)] stabilizes, you'll still need a feature flag to avoid compiling your benchmarks in release.

[nvzqz]

This is quite unfortunate, but thanks @jdidion for finding a decent workaround! I'll leave this issue open until this works as expected.

[dvdplm]

@jdidion I ran into the visibility crate the other day, which could be helpful to make your workaround less verbose.

@nvzqz I did some debugging on this issue; I didn't get super far, but I did manage to prove to myself that the benchmark does make it into the __mod_init_func section.
As I'm sure you are aware there has been plenty of discussion around the #[used] attribute and I was worried that we needed the new #[used(linker)] variant to prevent the compiler from optimizing out divan. This is not the case. There must be something else stopping macOS from actually calling the code before main, but I have not figured out why.

[nvzqz]

It appears that this issue is caused by the compiler completely discarding the crate where internal benchmarks are defined. If no items are used, then none of the crate's symbols appear in the final binary. I verified this with otool -L.

I attempted some approaches for internal benchmark discovery, but they didn't work:

• #[linkme::distributed_slice] produces a zero-sized slice.
• Calling getsectiondata returns null for every Mach-O image header (obtained from _dyld_image_count + _dyld_get_image_header)

However, once I use any item from the crate, then all of the crate's other symbols are now also included. I'm hoping that rust-lang/testing-devex-team#3 might fix this. cc @epage

In the meantime, the RUSTFLAGS environment variable can be set to either of the following as a workaround:

• -Clinker=rust-lld
• -Clink-arg=-all_load

See rust-lang/rust#133491 for upstream discussion of this issue.

[nvzqz]

@Fraser999 the repro code you provided now works under Rust nightly from the fix in rust-lang/rust#133832.

However, this issue isn't entirely resolved since removing the extern crate divan_example; import makes it stop working.