Optimize validator binaries with LTO

[crystalin]

Currently, the polkadot binary is generated with the default release profile. This allow a good binary with a decent compilation time.
However using compilations options CARGO_PROFILE_RELEASE_LTO=true RUSTFLAGS="-C codegen-units=1" cargo build --release , it provides a much more efficient binary, 200-400% cpu perf (at the cost of twice longer to compile).

I suggest to not include it in the Cargo.toml to avoid people developing on it to get a longer compilation time. In Moonbeam, we include it only in the CI for the release binary and we added those flags in the documentation for operators building their own binary.

I believe it is important to start optimizing the validators as it is becoming a bottleneck to process the parachain blocks in time.

[ordian]

cc @s3krit

[bkchr]

And cc @chevdor

Please realize this already for the 0.9.13 release.

I would also propose that we push a commit to each release branch that adds lto = true to the Cargo.toml file. As @crystalin said, this is relative expensive and we don't want this on master, but I don't see any reason why not add this to the release branches. The positive effect of this would be that everybody, who builds the binary on its own, would build with lto by default.

[ordian]

Do you know how much does RUSTFLAGS="-C codegen-units=1" bring on top of LTO?

We can also add codegen-units = 1 in https://doc.rust-lang.org/cargo/reference/profiles.html#release along with lto = true.

[burdges]

Just fyi, RUSTFLAGS="-C codegen-units=1" changes trait object pointer equality, only in a beneficial way of course. Yet, it's conceivable someone writes code that depends upon codegen-units=1 and breaks when compiled otherwise. I don't think this hurts anything, and running production on codegen-units=1 and tests without is probably ideal, but just fyi..

[crystalin]

> Do you know how much does RUSTFLAGS="-C codegen-units=1"` bring on top of LTO?

We can also add codegen-units = 1 in https://doc.rust-lang.org/cargo/reference/profiles.html#release along with lto = true.

codegen-units=1 is not bringing any improvement, it can be skipped

[crystalin]

FYI, other optimizations that we tried:

• PGO (didn't give good result, but it is hard to setup properly)
• target-cpu=native (Good result for recent CPU, but can't be applied to generic release. In Moonbeam we provide 3 binaries in release, moonbeam, moonbeam-skylake, moonbeam-znver3 to satisfy 80% of our collators)

[s3krit]

The compilation flags provided in the OP (CARGO_PROFILE_RELEASE_LTO=true RUSTFLAGS="-C codegen-units=1" cargo build --release) have been integrated into our internal build system for building the release binaries, packages, docker images, etc. Cheers!

[bkchr]

Latest rust release is bringing exactly what we want here! Custom profiles: https://blog.rust-lang.org/2021/12/02/Rust-1.57.0.html#cargo-support-for-custom-profiles

[ggwpez]

The compilation flags provided in the OP (CARGO_PROFILE_RELEASE_LTO=true RUSTFLAGS="-C codegen-units=1" cargo build --release) have been integrated into our internal build system for building the release binaries, packages, docker images, etc. Cheers!

So the binaries are using the flags, but the benchmarks are run without, right? @s3krit