benchmark runner is passed `--bench` even when `harness = false`

[bluss]

I'm experimenting with benchmarking for Rust stable. One way to tie into cargo is simply to use the benchmark facility:

[[bench]]
name = "benchmarks"
harness = false

This compiles and runs, using a dev-dependency in the benchmark runner and so on. The only unexpected part is that it is passed --bench as a command line argument (and to add to that, it's not the first argument but the last). I'd expect it does not use --bench when harness = false.

[alexcrichton]

Sounds reasonable to me! Unfortunately this won't be trivial to implement in Cargo, but the info's all there in one way or another, just need to connect the dots.

[epage]

Could you clarify what the downside is to --bench being passed in?

At this point, this seems like doing so would break backwards compatibility and I'd propose to the cargo team we close this. There might be ways to make a transition for this (already for other issues, I want to offer ways to control the protocol between cargo and harnesses) but that is moving in the direction of opting into more requirements on harnesses, not less, and any work in this direction would come at a cost of time and complexity.

[bluss]

It's a while since I worked with this, but I think the gist of it is that for harness = false it's a custom program that can have its own command line interface. I think the user should be able to define this interface, or at least the mandatory parts of it (enforced --bench) should be documented.

If any of that is now documented and so on, I don't know now though. 🙂

[weihanglo]

or at least the mandatory parts of it (enforced --bench) should be documented.

This sounds a preferable plan to me, though I found it already documented: https://doc.rust-lang.org/nightly/cargo/commands/cargo-bench.html#description

Cargo passes the --bench flag to the test harness to tell it to run only benchmarks.

I think we can add something like "regardless of whether the harness is libtest or a custom harness.", at the end of the sentence.