Proposal: change nightlies from ReleaseFast to ReleaseSafe+debug

[motiejus]

Currently releases and nightly builds on ziglang.org/download are compiled with ReleaseFast. This has two complications:

1. When we encounter compiler bugs, we need to rebuild Zig with debug symbols and (try to) reproduce the issue again.
2. ReleaseFast skips assertions. We may not know we hit a compiler bug that until long after compiling.

Users who want to build programs of others' should use the last-published version: that's fast and small. Users who are picking the nightlies know are coming for a ride; we can make that ride a wee bit safer for them. Therefore I suggest:

1. Compiled the nightlies (the ones that end up in https://ziglang.org/download) with -OReleaseSafe and do not strip.
2. The point releases should remain as-is -- -OReleaseFast -Dstrip.

Some immediate benefits and next steps:

1. We would have stack traces for each repro of zig prereq races as of 0.11.0-dev.1782+b52be973d #14815 instead of error: FileNotFound.
2. In macos-aarch64: segfault when running a CGo binary #14923: the binary would not have been created in the first place, since there was a good assertion. Since assertion was skipped, the binary was created and then segfaulted.
3. I am planning to propose a CI runner that uses zig cc as the C compiler to the Go team (pending last issue runtime/pprof: missing profile info for CGo code when compiled with CC="zig cc" golang/go#59466). We will quite certainly want them to run a debug build -- so the back-and-forth reporting can be much quicker due to stack traces and assertions.
4. If performance change is acceptably low, I would propose to use such build at my employer for all our C/C++ builds.

Depends on #15192

File Size impact

I compiled 0.11.0-dev.2257+f99b75360 (the version of today's zig-bootstrap) with the following flags:

1. baseline: what we have today, except xz -9e (I think currently we use -6). All archives below use this level.
2. safe-strip: -OReleaseSafe -Dstrip.
3. safe-debug: -OReleaseSafe. This is the one I propose for nightlies.
4. safe-zdebug: -OReleaseSafe --compress-debug-sections=zlib.

For linux-x86_64 build the archive increases from 38M to 51M.

$ ls -hsS1 zig-x86_64-linux-musl-baseline-*/bin/zig *.tar.xz | tac
 38M zig-x86_64-linux-musl-baseline-baseline.tar.xz
 39M zig-x86_64-linux-musl-baseline-safe-strip.tar.xz
 51M zig-x86_64-linux-musl-baseline-safe-debug.tar.xz
 55M zig-x86_64-linux-musl-baseline-safe-zdebug.tar.xz
132M zig-x86_64-linux-musl-baseline-safe-strip/bin/zig
133M zig-x86_64-linux-musl-baseline-baseline/bin/zig
174M zig-x86_64-linux-musl-baseline-safe-zdebug/bin/zig
222M zig-x86_64-linux-musl-baseline-safe-debug/bin/zig

[motiejus]

A related open question: should the tagged Zig versions have both options available for download: ReleaseFast (the big green default) and ReleaseSafe+Debug (hidden under a math test?).

Tagged Zig ReleaseSafe+Debug would be encouraged for third-party integrations. For instance, the CI workers of other languages that want to use Zig as the compiler/linker.

[motiejus]

Just compared the performance of ReleaseFast+strip and ReleaseSafe+Debug on test-zigcc. This test suite designed to stress-test as many invocations of zig cc as possible (in this case, 128) with a very, very simple C file.

The results are as follows:

ReleaseSafe + debug

Benchmark 1: bazel --batch  build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //...
  Time (mean ± σ):     21.302 s ±  0.754 s    [User: 67.155 s, System: 52.041 s]
  Range (min … max):   20.701 s … 22.199 s    5 runs

ReleaseFast + strip

Benchmark 1: bazel --batch  build --platforms=@zig_sdk//libc_aware/platform:linux_amd64_gnu.2.28 //...
  Time (mean ± σ):     20.466 s ±  0.442 s    [User: 62.760 s, System: 52.654 s]
  Range (min … max):   19.977 s … 21.076 s    5 runs

We can observe a consistent 4-5% performance loss for this test, which is notable, but not overly concerning.

[davidgmbb]

It would also be nice to have some assertions/feeback on LLVM side which doesn't result in a crash with no information. I don't know how much this will slow the compiler, given that LLVM is currently the major overhead.

[motiejus]

It would also be nice to have some assertions/feeback on LLVM side which doesn't result in a crash with no information. I don't know how much this will slow the compiler, given that LLVM is currently the major overhead.

That would necessitate a debug build of LLVM with debug symbols (so we get useful information from the crash), which, if I remember Andrew's anecdotes correctly, is on the order of gigabytes (over 10 gigabytes?).

[davidgmbb]

There is a flag LLVM_ENABLE_ASSERTIONS described in https://github.com/ziglang/zig/wiki/How-to-build-LLVM,-libclang,-and-liblld-from-source#release that seems to be more minimal than that, but I have not measured size or speed changes. When I built LLVM 16 recently, it was triggering an assert while the ReleaseFast version was not crashing. I assume we don't get many benefits of poor-quality failing asserts either, but it should be at least consider in my opinion.

[motiejus]

There is a flag LLVM_ENABLE_ASSERTIONS described in https://github.com/ziglang/zig/wiki/How-to-build-LLVM,-libclang,-and-liblld-from-source#release that seems to be more minimal than that, but I have not measured size or speed changes. When I built LLVM 16 recently, it was triggering an assert while the ReleaseFast version was not crashing. I assume we don't get many benefits of poor-quality failing asserts either, but it should be at least consider in my opinion.

Sounds like a good idea, but then even more performance testing is warranted. I agree we should consider it.

The proposal is "accepted", but I am not completely privy from which code the nightly releases are built. Does it come from zig-bootstrap? Should we change these flags in zig-bootstrap?

Anyhow, related: ziglang/zig-bootstrap#149

[andrewrk]

Indeed, the CI-provided builds are produced via zig-bootstrap. Perhaps another argument can be added to the build script for the optimization mode passed to zig build. I would like to keep the build script free of branching logic.

[motiejus]

Indeed, the CI-provided builds are produced via zig-bootstrap. Perhaps another argument can be added to the build script for the optimization mode passed to zig build. I would like to keep the build script free of branching logic.

What is your opinion on enabling LLVM_ENABLE_ASSERTIONS for the llvm build of zig-bootstrap?

Edit: replied during a random twitch stream that the slowdown is about 50%, which is unacceptable for the use case I propose.