Emitting "oom" doesn't mesh with the org.freedesktop.oom1.Manager
Killed() API contract, which defines "memory-used" and "memory-pressure"
as possible reasons. Consumers that key off reason thus will either lose
policy attribution or may reject the unknown value completely.
Plumb the reason through so it is visible to consumers.
When a cgroup is selected for termination, send varlink messages
to hooks registered in `/run/systemd/oomd.prekill-hooks/`.
oomd waits up to `PreKillHookTimeoutSec=` seconds for response
before proceeding with the kill.
Let's not leak details from src/shared and src/libsystemd into
src/basic, even though you can't actually do anything useful with
just forward declarations from src/shared.
The sd-forward.h header is put in src/libsystemd/sd-common as we
don't have a directory for shared internal headers for libsystemd
yet.
Let's also rename forward.h to basic-forward.h to keep things
self-explanatory.
This will allow units (scopes/slices/services) to override the default
systemd-oomd setting DefaultMemoryPressureDurationSec=.
The semantics of ManagedOOMMemoryPressureDurationSec= are:
- If >= 1 second, overrides DefaultMemoryPressureDurationSec= from oomd.conf
- If is empty, uses DefaultMemoryPressureDurationSec= from oomd.conf
- Ignored if ManagedOOMMemoryPressure= is not "kill"
- Disallowed if < 1 second
Note the corresponding dbus property is DefaultMemoryPressureDurationUSec
which is in microseconds. This is consistent with other time-based
dbus properties.
The ManagedOOMPreference property is only honored on cgroups which are
owned by root. This precludes anyone from setting ManagedOOMPreference
on cgroups managed by user managers.
Loosen this restriction in the following way: when processing a
monitored cgroup for kill candidates, honor the ManagedOOMPreference
setting if the monitored cgroup and cgroup candidate are owned by the
same user. This allows unprivileged users to configure
ManagedOOMPreference on their cgroups without affecting the kill
priority of ancestor cgroups.
N.B. that since swap kill operates globally to kill the largest
candidate, it is not appropriate to apply this logic to the swap kill
scenario. Therefore, the existing restriction on ManagedOOMPreference
will remain when calculating candidates for swap kill.
Add a new function, oomd_fetch_cgroup_oom_preference, to assist with
this new logic. To simplify things, move the `user.oomd_{avoid,omit}`
xattr reads to this function so that the xattr reads and uid checks are
performed all at once.
The calculation for used memory in oomd_system_context_acquire is given
by MemTotal - MemFree from /proc/meminfo. This is too strict of a
calculation because it does not consider memory that is still available
for starting new applictions without swapping (MemAvailable). As a
result, systemd-oomd can start to kill processes before it is necessary.
This is more apparent on systems with low swap space.
Instead, compute 'used' memory as MemTotal - MemAvailable in
oomd_system_context_acquire and procfs_memory_get (which is used by
oomd_cgroup_context_acquire). And, rename oomd_mem_free_below to
oomd_mem_available_below for clarity.
In some instances, particularly with swap on zram, swap used will be high
while there is still a lot of memory available. FB OOMD handles this by
thresholding kills to X% of total swap usage. Let's do the same thing here.
Anecdotally with these thresholds and my laptop which is exclusively swap
on zram I can sit at 0K / 4G swap free with most of memory free and
systemd-oomd doesn't kill anything.
Partially addresses aggressive kill behavior from
https://bugzilla.redhat.com/show_bug.cgi?id=1941170
Since this is only changed the first time the limit is hit (and remains
set as long as the pressure remains over), I changed the name to better
reflect that.
Keeps consistent with "last_had_mem_reclaim" which is actually updated
every time there is reclaim activity.
systemd-oomd only monitors and kills within a selected cgroup subtree
For memory pressure kills, this means it's unnecessary to get the
pgscan rate across all the monitored memory pressure cgroups.
The increase will show up whether we do a total sum or not, but since
we only care about the increase in the subtree we're about to target
for a kill, we can simplify the code a bit by not doing this total sum.
Improve the logging to only print if systemd-oomd killed something. And
also print which cgroup was targeted.
Demote general swap above/pressure above messages to debug.
[zjs: fix some issuelets found in review]
For pressure based killing we want to target who has the highest
increase in pgscan from the previous interval (vs. the previous logic
which used raw pgscan). This will prevent biasing towards long running
cgroups as mentioned in #19007.
oomd.conf has two parameters with fractionals: SwapUsedLimit= and
DefaultMemoryPressureLimit=, but one accepts permyriads, the other only
percentages, for no apparent reason. One carries the "Percent" in the
name, the other doesn't.
Let's clean this up: always accept permyriads, and drop the suffix,
given that it is misleading.
I figure we should internally try to focus on scaling everything
relative to UINT32_MAX, and if that isn't in the cards at least 10000,
but never permille nor percent unless there's a really really good
reason for it (e.g. interface defined by someone else).
There may be situations where a cgroup should be protected from killing
or deprioritized as a candidate. In FB oomd xattrs are used to bias oomd
away from supervisor cgroups and towards worker cgroups in container
tasks. On desktops this can be used to protect important units with
unpredictable resource consumption.
The patch allows systemd-oomd to understand 2 xattrs:
"user.oomd_avoid" and "user.oomd_omit". If systemd-oomd sees these
xattrs set to 1 on a candidate cgroup (i.e. while attempting to kill something)
AND the cgroup is owned by root, it will either deprioritize the cgroup as
a candidate (avoid) or remove it completely as a candidate (omit).
Usage is restricted to root owned cgroups to prevent situations where an
unprivileged user can set their own cgroups lower in the kill priority than
another user's (and prevent them from omitting their units from
systemd-oomd killing).
Chris Down
Matteo Croce
Daan De Meyer
Ryan Wilson
Yu Watanabe
Nick Rosbrook
Benjamin Berg
Anita Zhang
Lennart Poettering