This time, various commands with non-trivial requirements are handled.
So the series is shorter, but has a pattern of feature additions and
then those new features being used. The way that --help output is
formatted is changed a bit in various places. Those changes are
described in individual commits.
22 'static int help()' functions left after this.
We already have io.systemd.service.SetLogLevel. Add a getter so that
CLI tools with `log-level` subcommands can be more easily implemented
with varlink.
Bind this method to all services that already bind io.systemd.service.SetLogLevel.
Almost all programs are simple and have only a single command. When
printing options, let's just use the first COMMAND block found by default.
This means we can drop the explicit program-name argument and reduce
copy&paste boilerplate.
Accept -n as a short option for --dry-run in bootctl,
systemd-oomd, systemd-sysusers, and systemd-tmpfiles.
For systemd-repart, make -n equivalent to --dry-run=yes,
while keeping --dry-run=BOOL available.
Follow-up for: 2479f0bb09
We have two concepts: a list of files that should be extracted from an
intermediate build artifact to be used in a later build artifact, and for a
given artifact, a list "donors". The first list was called "extract", because
it is passed to the .extract_objects() meson function, and the second was
called "objects", because that is the meson parameter to specify pre-built
objects files (6350d2dbd9). But this naming is
confusing: we don't care about the 'extract' step, this is something internal
to the build machinery. And 'objects' is a very generic term.
Let's use 'export' for the stuff that is "exported" for other binaries to use,
and 'import' to say where to import from. Those terms are symmetrical and the
association between them should be intuitive. (If you think the terms are
actually assymetrical, there are precedents for confusing the import with the
import sources. E.g. in modern English, turkeys are called so because they were
imported from the Americas and guineafowls were imported from Africa via
Türkiye and all that foreign stuff is alike.)
The log message is informational only, same as the count of killed
processes, so we can safely use cgroup.kill when available instead
of manually recursing through the cgroup tree.
The log message is informational only, same as the count of killed
processes, so we can safely use cgroup.kill when available instead
of manually recursing through the cgroup tree.
test_oomd_cgroup_context_acquire_and_insert sets c1->pgscan = UINT64_MAX
and c1->last_had_mem_reclaim = 888 before re-inserting via
oomd_insert_cgroup_context(h1, h2, cgroup), then asserts that c2 carries
over last_had_mem_reclaim == 888.
The assumption was that, because c1->pgscan (which becomes c2->last_pgscan)
is UINT64_MAX and is therefore strictly greater than the freshly-acquired
c2->pgscan, oomd_pgscan_rate() in oomd_insert_cgroup_context() would
return 0 and leave last_had_mem_reclaim untouched. But the function
treats last_pgscan > pgscan as "cgroup was recreated", resets last_pgscan
to 0, and returns pgscan - 0 == pgscan. So when the kernel records any
non-zero pgscan on the live cgroup of the test unit, the rate is > 0 and
last_had_mem_reclaim is overwritten with now(CLOCK_MONOTONIC).
From src/oom/oomd-util.c:
uint64_t oomd_pgscan_rate(const OomdCGroupContext *c) {
[...]
last_pgscan = c->last_pgscan;
if (c->last_pgscan > c->pgscan) {
log_debug("Last pgscan %"PRIu64" greater than current pgscan %"PRIu64" for %s. Using last pgscan of zero.",
c->last_pgscan, c->pgscan, c->path);
last_pgscan = 0;
}
return c->pgscan - last_pgscan;
}
int oomd_insert_cgroup_context(Hashmap *old_h, Hashmap *new_h, const char *path) {
[...]
if (oomd_pgscan_rate(curr_ctx) > 0)
curr_ctx->last_had_mem_reclaim = now(CLOCK_MONOTONIC);
Under sanitizers the test can be slow enough that the kernel scans the
test service cgroup at least once between the two acquisitions, which
triggers the failure:
[ 1898.159455] test-oomd-util[2606]: /* test_oomd_cgroup_context_acquire_and_insert */
[ 1898.161247] test-oomd-util[2606]: Last pgscan 18446744073709551615 greater than current pgscan 2 for /system.slice/test-oomd-util.service. Using last pgscan of zero.
[ 1898.161330] test-oomd-util[2606]: src/oom/test-oomd-util.c:322: Assertion failed: Expected "c2->last_had_mem_reclaim == 888u", but 1898161250 != 888
(1898161250 matches the monotonic timestamp of the log line, confirming
that last_had_mem_reclaim was overwritten with now(CLOCK_MONOTONIC).)
The propagation of last_had_mem_reclaim is already covered
deterministically by TEST(oomd_update_cgroup_contexts_between_hashmaps),
which builds OomdCGroupContext structs directly with matching pgscan
values so that oomd_pgscan_rate() returns exactly 0. So just drop the
racy setup and assertion here.
Follow-up for df637ede7b
Fixes https://github.com/systemd/systemd/issues/38543
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
The test asserts that pgscan is 0, but under sanitizers this sometimes
fails and shows up as 1. We cannot control what the kernel scans, and
with sanitizers the runtime can be slow enough it's possible that the
kernel does a pass on the cgroup of the unit test.
Instead of asserting that it's 0, assert that it's between 0 and 9,
which seems a reasonable range.
Fixes https://github.com/systemd/systemd/issues/37710
I am really not a fan of full code lines passed to macros as parameters.
Let's get rid of the 3rd parameter of FOREACH_OPTION() hence:
1. Let's return errors just as a regular value (though a negative one),
that can be handled via a OPTION_ERROR case statement for the switch.
This normalizes handling of the error, just like any other event
returned by the option parser.
2. In order to avoid exploding the amount of boilerplate in each use
(that just propagates the error on OPTION_ERROR), let's then
introduce an explicit FOREACH_OPTION_OR_RETURN(), that returns from
the calling function on its own (and makes that clear in the name).
Together this cleans up, normalizes the logic and shortens the code.
It was requested to make the 'c', 'opt', and 'arg' params the same, i.e.
defined through the FOREACH_OPTION macro. But we can't do that easily,
because 'c' was defined in the for loop definition, and we can only
define variables of the same type in that way. Also, in some cases we
need only 'c', in other cases with need 'c' and 'arg, in some cases 'c'
and 'opt', and in other cases all three. We'd need to either
conditionalize or mark those variables with _unused_ to deal with
compiler warnings. But a different approach works quite nicely: add
state.opt and state.arg to show the current option and it's argument.
(The short names are picked on purpose to reduce verbosity since those
are used a lot.)
FAIL: check-pointer-deref.cocci found issues in /home/runner/work/systemd/systemd/src/oom/:
diff -u -p /home/runner/work/systemd/systemd/src/oom/oomctl.c /tmp/nothing/oomctl.c
--- /home/runner/work/systemd/systemd/src/oom/oomctl.c
+++ /tmp/nothing/oomctl.c
@@ -107,7 +107,6 @@ static int parse_argv(int argc, char *ar
break;
}
- *ret_args = option_parser_get_args(&state);
return 1;
}
Coccinelle check(s) failed. For each flagged dereference, either:
- Add assert(param)/ASSERT_PTR(param) at the top of the function (if the parameter must not be NULL)
- Add an if (param) guard before the dereference (if NULL is valid)
- Add POINTER_MAY_BE_NULL(param) if NULL is okay for param
Follow-up for 143af68ee1
Multiple callers of cg_get_keyed_attribute() follow the same pattern of
reading a single keyed attribute and then parsing it as uint64_t with
safe_atou64(). Add a helper that combines both steps.
Convert all existing single-key + uint64 call sites in cgtop, cgroup.c,
and oomd-util.c to use the new helper.
The fix in 8f1751a111 made me wonder if we could automatically detect
when pointers are accessed but when this might not be safe. Systemd
is already using a lot of `assert(dst)` and this change now forces
us to use them.
So this commit (ab)uses coccinelle to flag any pointer parameter
dereference not preceded by assert(param), ASSERT_PTR(param), or an
explicit NULL check. It adds integration into meson as a new "coccinelle"
test suite (just like clang-tidy) and is run in CI. The check is not
perfect but seems a reasonable heuristic.
For this RFC commit it is scoped to a subset, it excludes 25 dirs right
now and includes around 100. About 300 warnings left. Busywork that I am
happy to do if there is agreement that it is worth it.
With this in place we would have caught the bug from 8f1751a111 in CI:
```
FAIL: check-pointer-deref.cocci found issues in systemd/src/boot:
diff -u -p systemd/src/boot/measure.c /tmp/nothing/measure.c
--- systemd/src/boot/measure.c
+++ /tmp/nothing/measure.c
@@ -312,7 +312,6 @@ EFI_STATUS tpm_log_tagged_event(
if (err != EFI_SUCCESS)
return err;
- *ret_measured = true;
return EFI_SUCCESS;
}
```
This also adds a new POINTER_MAY_BE_NULL() for the cases when the
called function will do the NULL check (like `iovec_is_set()`).
We often have a pattern where the same verb function is used for
multiple actions. This leads to an antipattern where we figure out what
action needs to be taken from argv[0] multiple times: often once in
parse_argv() to figure out what options are allowed, then once again
implicitly in dispatch_verb(), and then again in the action verb itself.
Let's allow passing a parameter into the verb to simplify this.
This matches a pattern we have in conf-parser.h, where we have both
void *userdata (more global) and void *data (per-config item). Here,
I opted for uintptr_t userdata. It seems that most of the time we'll
want to just pass an enum value. This works OK with no casts. I also
tried a void* and union. In both cases, much more boilerplate is needed:
either a cast or a macro to help avoid compiler warnings. uintptr_t
seems generic enough to cover foreseeable usecases with no fuss.
This is a noop refactoring. See next commit for an example.
Let's say a user has two services with ManagedOOMMemoryPressure=kill,
perhaps a web server under system.slice and a batch job under
user.slice. Both exceed their pressure limits. On the previous timer
tick, oomd has already queued the web server's candidate for killing,
but the prekill hook has not yet responded, so the kill is still
pending.
In the code, monitor_memory_pressure_contexts_handler() iterates over
all pressure targets that have exceeded their limits. When it reaches
the web server target and calls oomd_cgroup_kill_mark(), which returns 0
because that cgroup is already queued. The code treats this the same as
a successful new kill: it resets the 15 second delay timer and returns
from the function, exiting the loop.
This loop is handled by SET_FOREACH and the iteration order is
hash-dependent. As such, if the web server target happens coincidentally
to be visited first, oomd never evaluates the batch job target at all.
The effect is twofold:
1. oomd stalls for 15 seconds despite not having initiated any new kill.
That can unnecessarily delay further action to stem increases in
memory pressure. The delay exists to let stale pressure counters
settle after a kill, but no kill has happened here.
2. It non-deterministically skips pressure targets that may have
unqueued candidates, dangerously allowing memory pressure to persist
for longer than it should.
Fix this by skipping cgroups that are already queued so the loop
proceeds to try other pressure targets. We should only delay when a new
kill mark is actually created.
Consider a workload slice with several sibling cgroups. Imagine that one
of those cgroups is removed between the moment oomd enumerates the
directory and the moment it reads memory.oom.group. This is actually
relatively plausible under the high memory pressure conditions where
oomd is most needed.
In this case, the failed read prompts us to `return 0`, which exits the
entire enumeration loop in recursively_get_cgroup_context(). As a
result, all remaining sibling cgroups are silently dropped from the
candidate list for that monitoring cycle.
The effect is that oomd can fail to identify and kill the actual
offending cgroup, allowing memory pressure to persist until a subsequent
cycle where the race doesn't occur.
Fix this by instead proceeding to evaluate further sibling cgroups.
Let's say a user has two services with ManagedOOMMemoryPressure=kill,
one for a web server under system.slice, and one for a batch job under
user.slice. The batch job is causing severe memory pressure, whereas the
web server's cgroup has no processes with significant pgscan activity.
In the code, monitor_memory_pressure_contexts_handler() iterates over
all pressure targets that have exceeded their limits. When
oomd_select_by_pgscan_rate() returns 0 (that is, no candidates) for a
target, we return from the entire SET_FOREACH loop instead of moving to
the next target. Since SET_FOREACH iteration order is hash-dependent, if
the web server target happens to be visited first, oomd will find no
kill candidates for it and exit the loop. The batch job target that is
actually slamming the machine will never even be evaluated, and can
continue to wreak havoc without any intervention.
The effect is that oomd non-deterministically and silently fails to kill
cgroups that it should actually kill, allowing memory pressure to
persist and dangerously build up on the machine.
The fix is simple, keep evaluating remaining targets when one does not
match.
Emitting "oom" doesn't mesh with the org.freedesktop.oom1.Manager
Killed() API contract, which defines "memory-used" and "memory-pressure"
as possible reasons. Consumers that key off reason thus will either lose
policy attribution or may reject the unknown value completely.
Plumb the reason through so it is visible to consumers.
oomd_cgroup_kill_mark() currently returns 0 on all non-error paths. But
the manager only logs that it marked for killing on `if (r > 0)`, which
is thus unreachable.
Changing it to `r >= 0` would also be wrong, because then we would log
on no-op paths.
So let's fix this by making the return value express what actually
happened:
- < 0: failure to queue the kill state
- 0: no new mark was created (already queued or dry-run)
- > 0: a new kill state was queued
oomd_cgroup_kill_mark() allocates a temporary OomdKillState and inserts
it into kill_states via set_ensure_put(). This is keyed by cgroup path.
When the same cgroup is already queued, set_ensure_put() dutifully
returns 0.
The function then returns with
_cleanup_(oomd_kill_state_removep) still armed, which eventually calls
oomd_kill_state_free().
oomd_kill_state_free() removes from kill_states by cgroup-path key, so
because this path already exists, it will remove the existing queued
kill state instead of just dropping the temporary object.
This is wrong and results in mistakenly drops the queued kill state on
duplicates.
This can happen when a cgroup is marked multiple times before the first
queued kill state is consumed. The result is lost kill-state tracking
and incorrect prekill/kill sequencing.
Handle r == 0 explicitly by freeing only the temporary object and
leaving the already queued state intact.
While looking at oomd behaviour in production I noticed that I always
get garbage cgroup paths for the Killed event. Looking more closely, I
noticed that while the signature is (string cgroup, string reason), we
currently erroneously pass the `OomdCGroupContext*` pointer itself as
the first argument to sd_bus_emit_signal(), rather than the ctx->path
string it contains.
The in-memory layout on affected machines in my case is:
struct OomdCGroupContext {
unsigned n_ref;
/* padding */
char *path;
/* ... */
}
...which explains the control characters, since they're garbage from
parsing n_ref, the path pointer, and later fields. At runtime, sd-bus
treats ctx as `const char *` and reads struct bytes as string data,
resulting in garbage being sent.
Pass ctx->path correctly so listeners receive the valid cgroup path.
When a cgroup is selected for termination, send varlink messages
to hooks registered in `/run/systemd/oomd.prekill-hooks/`.
oomd waits up to `PreKillHookTimeoutSec=` seconds for response
before proceeding with the kill.
This reverts commit 332bce5bd7.
The revert is needed because with the PreKill hook, oomd_cgroup_kill()
is not goint to really kill processes but it just creates the callbacks.
So the check is deferred to the real kill.
manager_varlink_init_resolve_hook and manager_varlink_init_userdb are very
similar, but one didn't set a description and the other one had an error
message which didn't print the offending path.
Define constants for the paths, and also change other similar constants
to have slightly shorter names. (It's an "address" and "path", but we don't
need to have both parts in the name, esp. that it makes the name very long.)
On OOM event, oomd tries to kill a cgroup until it succeedes.
The kill can fail with EPERM in case a pid is not killed, this leaves
the cgroup with only half of the processed killed.
This is unlikely but theoretically possible in a user namespace,
where systemd run as root inside the container and tries to kill a
cgroup with some PID from the host namespace.
To address this, send the SIG0 signal to all the processes to check
that we have privileges to kill them.
In all cases, everything that we list in 'extract', we also list in
'sources'. We can simplify things by automatically appending the first
list to the second.
In the listings, move 'extract' key right below 'sources', since now
they are both "sources", just with slightly different meanings.
felix h
Zbigniew Jędrzejewski-Szmek
Nick Rosbrook
dongshengyuan
Yu Watanabe
Christian Brauner
Luca Boccassi
Matteo Croce
Lennart Poettering
Yaping Li
Michael Vogt
Frantisek Sumsal
Chris Down
Daan De Meyer
Daan De Meyer
Zbigniew Jędrzejewski-Szmek