disable_coredump=1 does not disable coredump on systemd

[ecatmur]

Bugzilla Link	46452
Version	10.0
OS	Linux

Extended Description

(related: #​44214)

If /proc/sys/kernel/core_pattern is a pipe (e.g. '|/usr/bin/coredump <args...>') then setting RLIMIT_CORE to 0 does not actually disable coredumps; they are still generated and passed to the coredump helper, which will then send it to /dev/null (if written to respect RLIMIT_CORE).

Per systemd/systemd#2643 (comment) the solution is to set RLIMIT_CORE=1 (!) which will fully disable core dumping (no core is generated, the coredump helper is not invoked, and si_code is CLD_KILLED not CLD_DUMPED).

[ecatmur]

Suggested change:

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
index f920172c06d..d68c97e3263 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
@@ -108,7 +108,11 @@ static void setlim(int res, rlim_t lim) {

void DisableCoreDumperIfNecessary() {
if (common_flags()->disable_coredump) {
+#if SANITIZER_LINUX

• setlim(RLIMIT_CORE, std::max<rlim_t>(getlim(RLIMIT_CORE), 1));
  +#else
  setlim(RLIMIT_CORE, 0);
  +#endif
  }
  }

[EugeneZelenko]

Not merged yet.