Commits suicide

[Stebalien]

When I start running out of memory, earlyoom starts repeatedly killing itself (restarted by systemd).

OS: Arch Linux

Log Output:

earlyoom 1.5
Preferring to kill process names that match regex '(^|/)(gopls|rls)$'
Will avoid killing process names that match regex '(^|/)(emacs|emacsclient|X|systemd|zoom)$'
Could not set priority: Permission denied. Continuing anyway
Could not set oom_score_adj: Permission denied. Continuing anyway
mem total: 15898 MiB, swap total:    0 MiB
sending SIGTERM when mem <=  5.00% and swap <= 10.00%,
        SIGKILL when mem <=  2.50% and swap <=  5.00%
mem avail:   342 of 15898 MiB ( 2.16%), swap free:    0 of    0 MiB ( 0.00%)
low memory! at or below SIGKILL limits: mem  2.50%, swap  5.00%
sending SIGKILL to process 291787 uid 61876 "earlyoom": badness 0, VmRSS 0 MiB

Options:

/usr/bin/earlyoom -p -m 5 -r 0 --prefer (^|/)(gopls|rls)$ --avoid (^|/)(emacs|emacsclient|X|systemd|zoom)$

Systemd Unit:

[Unit]
Description=Early OOM Daemon
Documentation=man:earlyoom(1) https://github.com/rfjakob/earlyoom

[Service]
EnvironmentFile=-/etc/default/earlyoom
ExecStart=/usr/bin/earlyoom $EARLYOOM_ARGS
# Run as an unprivileged user with random user id
DynamicUser=true
# Allow killing processes and calling mlockall()
AmbientCapabilities=CAP_KILL CAP_IPC_LOCK
# We don't need write access anywhere
ProtectSystem=strict
# We don't need /home at all, make it inaccessible
ProtectHome=true
# earlyoom never exits on it's own, so have systemd
# restart it should it get killed for some reason.
Restart=always

[Install]
WantedBy=multi-user.target

[hakavlad]

Could you provide output (tasks list) with -d option? Did you start also ananicy daemon?

[Stebalien]

Did you start also ananicy daemon?

No.

Could you provide output (tasks list) with -d option?

Well, that explains it. earlyoom is running as a dynamic user and isn't in the proc group so it can't see other processes...

earlyoom 1.5
Preferring to kill process names that match regex '(^|/)(gopls|rls)$'
Will avoid killing process names that match regex '(^|/)(emacs|emacsclient|X|systemd|zoom)$'
Could not set priority: Permission denied. Continuing anyway
Could not set oom_score_adj: Permission denied. Continuing anyway
mem total: 15898 MiB, swap total:    0 MiB
sending SIGTERM when mem <=  5.00% and swap <= 10.00%,
        SIGKILL when mem <=  2.50% and swap <=  5.00%
dry-running kill_largest_process()...
pid 322425: badness   0 vm_rss     580 uid 61876 "earlyoom" <--- new victim
selecting victim took 0.408 ms
sending 0 (no-op signal) to process 322425 uid 61876 "earlyoom": badness 0, VmRSS 0 MiB
mem avail:   236 of 15898 MiB ( 1.49%), swap free:    0 of    0 MiB ( 0.00%)
low memory! at or below SIGKILL limits: mem  2.50%, swap  5.00%
pid 322425: badness   0 vm_rss     580 uid 61876 "earlyoom" <--- new victim
selecting victim took 0.332 ms
sending SIGKILL to process 322425 uid 61876 "earlyoom": badness 0, VmRSS 0 MiB

I'll file a bug upstream. Thanks!

[hakavlad]

Did you mount proc with hidepid=2?

Do you know how to fix the problem?

[hakavlad]

@rfjakob maybe this should be documented in FAQ.

[Stebalien]

@hakavlad yes. Sorry, I should have cross-linked the upstream bug report: https://bugs.archlinux.org/task/66134

The solution is to add SupplementaryGroups=proc to the systemd service unit. I see that the unit lives in this package, but I'm not sure what will go wrong if SupplementaryGroups=proc is specified and the group doesn't exist.

[hakavlad]

see also https://github.com/rfjakob/earlyoom/wiki/proc-hidepid

[rfjakob]

The situation is now handled gracefully via 571433b .