Let's say a user has two services with ManagedOOMMemoryPressure=kill,
perhaps a web server under system.slice and a batch job under
user.slice. Both exceed their pressure limits. On the previous timer
tick, oomd has already queued the web server's candidate for killing,
but the prekill hook has not yet responded, so the kill is still
pending.
In the code, monitor_memory_pressure_contexts_handler() iterates over
all pressure targets that have exceeded their limits. When it reaches
the web server target and calls oomd_cgroup_kill_mark(), which returns 0
because that cgroup is already queued. The code treats this the same as
a successful new kill: it resets the 15 second delay timer and returns
from the function, exiting the loop.
This loop is handled by SET_FOREACH and the iteration order is
hash-dependent. As such, if the web server target happens coincidentally
to be visited first, oomd never evaluates the batch job target at all.
The effect is twofold:
1. oomd stalls for 15 seconds despite not having initiated any new kill.
That can unnecessarily delay further action to stem increases in
memory pressure. The delay exists to let stale pressure counters
settle after a kill, but no kill has happened here.
2. It non-deterministically skips pressure targets that may have
unqueued candidates, dangerously allowing memory pressure to persist
for longer than it should.
Fix this by skipping cgroups that are already queued so the loop
proceeds to try other pressure targets. We should only delay when a new
kill mark is actually created.
Consider a workload slice with several sibling cgroups. Imagine that one
of those cgroups is removed between the moment oomd enumerates the
directory and the moment it reads memory.oom.group. This is actually
relatively plausible under the high memory pressure conditions where
oomd is most needed.
In this case, the failed read prompts us to `return 0`, which exits the
entire enumeration loop in recursively_get_cgroup_context(). As a
result, all remaining sibling cgroups are silently dropped from the
candidate list for that monitoring cycle.
The effect is that oomd can fail to identify and kill the actual
offending cgroup, allowing memory pressure to persist until a subsequent
cycle where the race doesn't occur.
Fix this by instead proceeding to evaluate further sibling cgroups.
Let's say a user has two services with ManagedOOMMemoryPressure=kill,
one for a web server under system.slice, and one for a batch job under
user.slice. The batch job is causing severe memory pressure, whereas the
web server's cgroup has no processes with significant pgscan activity.
In the code, monitor_memory_pressure_contexts_handler() iterates over
all pressure targets that have exceeded their limits. When
oomd_select_by_pgscan_rate() returns 0 (that is, no candidates) for a
target, we return from the entire SET_FOREACH loop instead of moving to
the next target. Since SET_FOREACH iteration order is hash-dependent, if
the web server target happens to be visited first, oomd will find no
kill candidates for it and exit the loop. The batch job target that is
actually slamming the machine will never even be evaluated, and can
continue to wreak havoc without any intervention.
The effect is that oomd non-deterministically and silently fails to kill
cgroups that it should actually kill, allowing memory pressure to
persist and dangerously build up on the machine.
The fix is simple, keep evaluating remaining targets when one does not
match.
These were introduced as part of the effort of sd-executor
worker pool (#29566), which never landed due to unsignificant
performance improvement. Let's just remove the unused
helpers. If that work ever gets resurrected they can be
restored from this commit pretty easily.
Emitting "oom" doesn't mesh with the org.freedesktop.oom1.Manager
Killed() API contract, which defines "memory-used" and "memory-pressure"
as possible reasons. Consumers that key off reason thus will either lose
policy attribution or may reject the unknown value completely.
Plumb the reason through so it is visible to consumers.
Emitting "oom" doesn't mesh with the org.freedesktop.oom1.Manager
Killed() API contract, which defines "memory-used" and "memory-pressure"
as possible reasons. Consumers that key off reason thus will either lose
policy attribution or may reject the unknown value completely.
Plumb the reason through so it is visible to consumers.
namespace_enter() now does a is_our_namespace() check, which requires
/proc on older kernels, which is not available anymore after we call
do_wipe_fully_visible_api_fs() in wipe_fully_visible_api_fs().
Let's just call setns() instead as namespace_enter() is overkill to
enter a single namespace anyway.
The default behavior is to sync repository metadata for every image
that does not have a cache and we recently changed behavior to
invalidate
all cached images whenever we decide the repository metadata needs to
be resynced.
In systemd we have two images that are not cached because they use
BaseTrees=
hence set CacheOnly=metadata to explicitly indicate these two images
should never cause a repository metadata if resync even though they are
not cached.
The default behavior is to sync repository metadata for every image
that does not have a cache and we recently changed behavior to invalidate
all cached images whenever we decide the repository metadata needs to
be resynced.
In systemd we have two images that are not cached because they use BaseTrees=
hence set CacheOnly=metadata to explicitly indicate these two images
should never cause a repository metadata if resync even though they are
not cached.
* 66d51024b7 man: Update caching section
* 4eac60f168 Remove all cached images if repository metadata will be synced
* 025c6c0150 Move Incremental= to inherited settings in docs
* 427970d8fd Make MakeScriptsExecutable= a multiversal setting
* 53bd2da6fe Look at all CacheOnly= settings to determine if we need to sync metadata
* 114ae558ef config / qemu: add Console=headless
There's no need to pass in a boolean to decide whether we use
namespace_enter_delegated() or not. Instead, we can just check if we
have CAP_SYS_ADMIN in our own user namespace. If we don't, then we have
to insist on a child user namespace being passed in and we have to enter
it first to get CAP_SYS_ADMIN as without CAP_SYS_ADMIN we wouldn't be
able
to call setns() in the first place. If we do have CAP_SYS_ADMIN, we can
always enter the other namespaces first before entering the user
namespace.
Additionally, we don't fail anymore if we can't reset the UID/GID since
a
root user might not always be available in every user namespace we might
enter.
This makes it possible to have DNS requests for certain domains routed
differently than normal requests, which is for instance useful when
using policy routing to route traffic over a VPN but DNS requests for
the VPN endpoint itself, should be routed differently.
It doesn't make much sense to configure a firewall mark at the level of
a network interface, but at the level of a DNS delegate it can be very
useful.
Also, this makes interfaces stored in OrderedSet rather than Set,
so interfaces becomes up/down in the specified order.
This also makes when multiple interfaces are specified, networkctl tries
to up/down all specfied interfaces even if some of them are failed.
Fixes#40017.
There's no need to pass in a boolean to decide whether we use
namespace_enter_delegated() or not. Instead, we can just check if we
have CAP_SYS_ADMIN in our own user namespace. If we don't, then we have
to insist on a child user namespace being passed in and we have to enter
it first to get CAP_SYS_ADMIN as without CAP_SYS_ADMIN we wouldn't be able
to call setns() in the first place. If we do have CAP_SYS_ADMIN, we can
always enter the other namespaces first before entering the user namespace.
Additionally, we don't fail anymore if we can't reset the UID/GID since a
root user might not always be available in every user namespace we might
enter.
Follow-up for bb1ef2edf7
The commit introduced a new "metrics" varlink server, but for
user scope stuff it is not bound anywhere. The copy-pasted
"fresh" handling for deserialization is also essentially
meaningless as metrics_setup_varlink_server() doesn't even report
whether the varlink server is fresh (let alone that no serialization
is being done at all right now). Moreover, currently the event
priority is hardcoded, while event loop and associated priority
assignment ought to be subject to each daemon.
While fixing the mentioned issues I took the chance to restructure
the existing code a bit for readability. Note that serialization
for the metrics server is still missing - it will be tackled
in subsequent commits.
Note that this actually matters: we might otherwise clobber
the logging fds reopened when rearranging fd '3'.
While at it, avoid logging from library functions.
If the IDL declares the method requires 'more' yet the call doesn't
have it set, varlink_idl_validate_method_call() should have rejected
it and the callback shouldn't be reached.
Chris Down
Mike Yuan
Yu Watanabe
Daan De Meyer
Ivan Kruglov
nikstur
r-vdp
Zbigniew Jędrzejewski-Szmek