The systemd-oomd-defaults rpm installed on Fedora/CentOS images via
mkosi/mkosi.conf.d/centos-fedora/mkosi.conf ships
/usr/lib/systemd/system/system.slice.d/10-oomd-per-slice-defaults.conf:
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=80%
https://gitlab.com/redhat/centos-stream/rpms/systemd/-/blob/c10s/10-oomd-per-slice-defaults.conf
In a recent flaky CI run on centos-10, the following sequence was
recorded in test/journal/TEST-55-OOMD-1.journal:
[ 77.740597] systemd-oomd[659]: Memory pressure for /system.slice is 85.03% > 80.00% for > 2s with reclaim activity
[ 77.745326] TEST-55-OOMD.sh[1292]: + journalctl --sync
[ 77.749125] systemd-oomd[659]: Considered 19 cgroups for killing, top candidates were:
[ 77.749128] systemd-oomd[659]: Path: /system.slice/systemd-journald.service
[ 77.749134] systemd-oomd[659]: Current Memory Usage: 55.5M
[ 77.749597] systemd-oomd[659]: oomd attempting to kill 407 with KILL
[ 77.749670] systemd-oomd[659]: Marked /system.slice/systemd-journald.service for killing due to memory pressure for /system.slice being 85.03% > 80.00% for > 2s with reclaim activity
[ 77.752252] systemd[1]: systemd-journald.service: systemd-oomd killed 1 process(es) in this unit.
[ 77.752293] systemd[1]: systemd-journald.service: Changed running -> stop-sigterm
[ 329.922401] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
PID 1292 (the test script) never logged anything after the
`journalctl --sync` line at [77.745326], and the unit stayed in
stop-sigterm until [329.922401].
Override /system.slice to ManagedOOMMemoryPressure=auto in the test
setup so the test does not rely on whichever per-slice oomd defaults
the distro happens to ship.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Other places that read credentials filer out ENXIO:
$ journalctl -b -u systemd-imds-early-network.service -o short-monotonic --no-hostname --no-pager -p warning
[ 5.477675] systemd-imdsd[473]: Failed to read credential 'imds.vendor', ignoring: No such device or address
[ 5.477687] systemd-imdsd[473]: Failed to read credential 'imds.vendor_token', ignoring: No such device or address
[ 5.477693] systemd-imdsd[473]: Failed to read credential 'imds.refresh_header_name', ignoring: No such device or address
[ 5.477699] systemd-imdsd[473]: Failed to read credential 'imds.data_url', ignoring: No such device or address
[ 5.477704] systemd-imdsd[473]: Failed to read credential 'imds.data_url_suffix', ignoring: No such device or address
[ 5.477710] systemd-imdsd[473]: Failed to read credential 'imds.token_header_name', ignoring: No such device or address
[ 5.477714] systemd-imdsd[473]: Failed to read credential 'imds.extra_header', ignoring: No such device or address
[ 5.477720] systemd-imdsd[473]: Failed to read credential 'imds.extra_header2', ignoring: No such device or address
[ 5.477730] systemd-imdsd[473]: Failed to read credential 'imds.extra_header3', ignoring: No such device or address
[ 5.477736] systemd-imdsd[473]: Failed to read credential 'imds.extra_header4', ignoring: No such device or address
[ 5.477742] systemd-imdsd[473]: Failed to read credential 'imds.extra_header5', ignoring: No such device or address
[ 5.477747] systemd-imdsd[473]: Failed to read credential 'imds.extra_header6', ignoring: No such device or address
[ 5.477751] systemd-imdsd[473]: Failed to read credential 'imds.extra_header7', ignoring: No such device or address
[ 5.477757] systemd-imdsd[473]: Failed to read credential 'imds.extra_header8', ignoring: No such device or address
[ 5.477763] systemd-imdsd[473]: Failed to read credential 'imds.extra_header9', ignoring: No such device or address
[ 5.477768] systemd-imdsd[473]: Failed to read credential 'imds.extra_header10', ignoring: No such device or address
[ 5.477774] systemd-imdsd[473]: Failed to read credential 'imds.extra_header11', ignoring: No such device or address
[ 5.477779] systemd-imdsd[473]: Failed to read credential 'imds.extra_header12', ignoring: No such device or address
[ 5.477784] systemd-imdsd[473]: Failed to read credential 'imds.extra_header13', ignoring: No such device or address
[ 5.477789] systemd-imdsd[473]: Failed to read credential 'imds.extra_header14', ignoring: No such device or address
[ 5.477799] systemd-imdsd[473]: Failed to read credential 'imds.extra_header15', ignoring: No such device or address
[ 5.477804] systemd-imdsd[473]: Failed to read credential 'imds.extra_header16', ignoring: No such device or address
[ 5.477810] systemd-imdsd[473]: Failed to read credential 'imds.extra_header17', ignoring: No such device or address
[ 5.477815] systemd-imdsd[473]: Failed to read credential 'imds.extra_header18', ignoring: No such device or address
[ 5.477820] systemd-imdsd[473]: Failed to read credential 'imds.extra_header19', ignoring: No such device or address
[ 5.477825] systemd-imdsd[473]: Failed to read credential 'imds.extra_header20', ignoring: No such device or address
[ 5.477830] systemd-imdsd[473]: Failed to read credential 'imds.extra_header21', ignoring: No such device or address
[ 5.477835] systemd-imdsd[473]: Failed to read credential 'imds.extra_header22', ignoring: No such device or address
[ 5.477840] systemd-imdsd[473]: Failed to read credential 'imds.extra_header23', ignoring: No such device or address
[ 5.477845] systemd-imdsd[473]: Failed to read credential 'imds.extra_header24', ignoring: No such device or address
[ 5.477851] systemd-imdsd[473]: Failed to read credential 'imds.extra_header25', ignoring: No such device or address
[ 5.477856] systemd-imdsd[473]: Failed to read credential 'imds.extra_header26', ignoring: No such device or address
[ 5.477865] systemd-imdsd[473]: Failed to read credential 'imds.extra_header27', ignoring: No such device or address
[ 5.477869] systemd-imdsd[473]: Failed to read credential 'imds.extra_header28', ignoring: No such device or address
[ 5.477874] systemd-imdsd[473]: Failed to read credential 'imds.extra_header29', ignoring: No such device or address
[ 5.477881] systemd-imdsd[473]: Failed to read credential 'imds.extra_header30', ignoring: No such device or address
[ 5.477885] systemd-imdsd[473]: Failed to read credential 'imds.extra_header31', ignoring: No such device or address
[ 5.477890] systemd-imdsd[473]: Failed to read credential 'imds.extra_header32', ignoring: No such device or address
[ 5.477896] systemd-imdsd[473]: Failed to read credential 'imds.extra_header33', ignoring: No such device or address
[ 5.477901] systemd-imdsd[473]: Failed to read credential 'imds.extra_header34', ignoring: No such device or address
[ 5.477906] systemd-imdsd[473]: Failed to read credential 'imds.extra_header35', ignoring: No such device or address
[ 5.477913] systemd-imdsd[473]: Failed to read credential 'imds.extra_header36', ignoring: No such device or address
[ 5.477917] systemd-imdsd[473]: Failed to read credential 'imds.extra_header37', ignoring: No such device or address
[ 5.477922] systemd-imdsd[473]: Failed to read credential 'imds.extra_header38', ignoring: No such device or address
[ 5.477931] systemd-imdsd[473]: Failed to read credential 'imds.extra_header39', ignoring: No such device or address
[ 5.477935] systemd-imdsd[473]: Failed to read credential 'imds.extra_header40', ignoring: No such device or address
[ 5.477940] systemd-imdsd[473]: Failed to read credential 'imds.extra_header41', ignoring: No such device or address
[ 5.477945] systemd-imdsd[473]: Failed to read credential 'imds.extra_header42', ignoring: No such device or address
[ 5.477951] systemd-imdsd[473]: Failed to read credential 'imds.extra_header43', ignoring: No such device or address
[ 5.477956] systemd-imdsd[473]: Failed to read credential 'imds.extra_header44', ignoring: No such device or address
[ 5.477962] systemd-imdsd[473]: Failed to read credential 'imds.extra_header45', ignoring: No such device or address
[ 5.477967] systemd-imdsd[473]: Failed to read credential 'imds.extra_header46', ignoring: No such device or address
[ 5.477972] systemd-imdsd[473]: Failed to read credential 'imds.extra_header47', ignoring: No such device or address
[ 5.477978] systemd-imdsd[473]: Failed to read credential 'imds.extra_header48', ignoring: No such device or address
[ 5.477982] systemd-imdsd[473]: Failed to read credential 'imds.extra_header49', ignoring: No such device or address
[ 5.477988] systemd-imdsd[473]: Failed to read credential 'imds.extra_header50', ignoring: No such device or address
[ 5.477995] systemd-imdsd[473]: Failed to read credential 'imds.extra_header51', ignoring: No such device or address
[ 5.477999] systemd-imdsd[473]: Failed to read credential 'imds.extra_header52', ignoring: No such device or address
[ 5.478022] systemd-imdsd[473]: Failed to read credential 'imds.extra_header53', ignoring: No such device or address
[ 5.478028] systemd-imdsd[473]: Failed to read credential 'imds.extra_header54', ignoring: No such device or address
[ 5.478033] systemd-imdsd[473]: Failed to read credential 'imds.extra_header55', ignoring: No such device or address
[ 5.478039] systemd-imdsd[473]: Failed to read credential 'imds.extra_header56', ignoring: No such device or address
[ 5.478044] systemd-imdsd[473]: Failed to read credential 'imds.extra_header57', ignoring: No such device or address
[ 5.478050] systemd-imdsd[473]: Failed to read credential 'imds.extra_header58', ignoring: No such device or address
[ 5.478055] systemd-imdsd[473]: Failed to read credential 'imds.extra_header59', ignoring: No such device or address
[ 5.478060] systemd-imdsd[473]: Failed to read credential 'imds.extra_header60', ignoring: No such device or address
[ 5.478066] systemd-imdsd[473]: Failed to read credential 'imds.extra_header61', ignoring: No such device or address
[ 5.478071] systemd-imdsd[473]: Failed to read credential 'imds.extra_header62', ignoring: No such device or address
[ 5.478079] systemd-imdsd[473]: Failed to read credential 'imds.extra_header63', ignoring: No such device or address
[ 5.478086] systemd-imdsd[473]: Failed read IPv4 address from credential 'imds.address_ipv4', ignoring: No such device or address
[ 5.478091] systemd-imdsd[473]: Failed read IPv6 address from credential 'imds.address_ipv6', ignoring: No such device or address
[ 5.478097] systemd-imdsd[473]: Failed to read credential 'imds.key_hostname', ignoring: No such device or address
[ 5.478102] systemd-imdsd[473]: Failed to read credential 'imds.key_region', ignoring: No such device or address
[ 5.478107] systemd-imdsd[473]: Failed to read credential 'imds.key_zone', ignoring: No such device or address
[ 5.478113] systemd-imdsd[473]: Failed to read credential 'imds.key_ipv4_public', ignoring: No such device or address
[ 5.478119] systemd-imdsd[473]: Failed to read credential 'imds.key_ipv6_public', ignoring: No such device or address
[ 5.478123] systemd-imdsd[473]: Failed to read credential 'imds.key_ssh_key', ignoring: No such device or address
[ 5.478128] systemd-imdsd[473]: Failed to read credential 'imds.key_userdata', ignoring: No such device or address
[ 5.478134] systemd-imdsd[473]: Failed to read credential 'imds.key_userdata_base', ignoring: No such device or address
[ 5.478137] systemd-imdsd[473]: Failed to read credential 'imds.key_userdata_base64', ignoring: No such device or address
In https://bugzilla.redhat.com/show_bug.cgi?id=2481466, it is reported
that systemd-imds-early-network.service fails on AWS EC2 instance with:
> No IMDS endpoint information provided or detected, cannot operate.
The service will be started on various systems where it doesn't have
enough configuration to run. The systemd-imds-generator only does some
superficial checks, so the tool should not mark the boot as failed
it turns out that IMDS is not supported.
Since a62cd5a153, when an invalid property
is being added, device_add_property() refuses it with -EINVAL.
Before the commit, the function could fail with only -ENOMEM, and if it fails,
processing udev rules was aborted.
Let's not hard fail when udev rules or udev builtins try to add an invalid
property, and let's just log and ignore the failure.
Follow-up for a62cd5a153.
Fixes#41339 and #41296.
* 9cb09470c9 Version 261~rc1
* 7d16ad00bd Fix compilation with openssl 4.0
* 3770fa7c92 Revert "Use dlopen-notes to automatically generate library Recommends/Requires/Suggests"
* 8b3d1726f5 Add back various explit Requires
* d8abb91805 Drop scriptlet workarounds for upgrades from versions < 247
* ee6d0b9d27 Use dlopen-notes to automatically generate library Recommends/Requires/Suggests
* 32a2038627 spec: drop duplicated line
* ab743db0fe Downgrade kbd dependency to Recommends
* 400494ada4 Update conditionals for RHEL 11
This test often fails because the VM reboots (eg: kernel panic) and the
previous state is left unclean. Do the cleanups at the beginning too,
to try and reduce flakiness.
After looking at the unit, I'm not sure if systemd-imds-import.service
is supposed to run in the host system or not. But if it is supposed to
only run in the initrd, then the static condition in the unit file gives
as the worst behaviour: the generator does not do any checks if we are
in the initrd or not, and if it enabled the unit, it'll influence the
transaction ordering (possibly causing loops or additional work) and
then the unit will be unconditionally skipped. So replace the static
condition by a check in the generator. If the user specifies
systemd.imds.import on the commandline, it'll be honoured also in the
host.
The imds services are (very-) early boot services, ordered before
sysinit.target. OTOH, network-online.target is something that is often
established very late [1]. In fact, we always recommended to _not_
depend on it for things that are ordered before the usual targets that
allow user logins (e.g. graphical.target). So we certainly cannot add
early-boot ordering for it. Currently systemd-imds-imports.service
is conditionalized to only run in the initrd. But if the unit is enabled
in the host system, it would affect the transaction there, so we should
drop this bogus ordering everywhere.
(I think this might have been added by mistake. After all,
systemd-imds-import.service itself shouldn't care about the network.)
This fixes boot ordering issues reported upstream and in Fedora [2, 3].
[1] https://systemd.io/NETWORK_ONLINE/#network-connectivity-has-been-established-network-onlinetarget
[2] https://github.com/systemd/systemd/pull/40980#discussion_r2949473028
[3] https://bugzilla.redhat.com/show_bug.cgi?id=2481304
This reverts commit a17efef137 ("test: try to detect SIGILL in stress-ng
and skip TEST-55-OOMD gracefully") and replaces it with a single
check to skip the test cases.
The previous check was not reliable as stress-ng can catch SIGILL
itself and exist with an error:
stress-ng[1068]: stress-ng: debug: [1068] caught SIGILL, address \
0x00005632f8330140 (ILL_ILLOPN)
stress-ng[1068]: stress-ng: debug: [1068] stress-ng: info: \
0x00005632f8330140:<62>71 fd 48 6f 2d 36 14 1c 00 c5 d1 ef ed 49 29
...
stress-ng[1053]: stress-ng: error: [1053] vm: [1061] terminated \
with an error, exit status=2 (stressor failed)
...
systemd[1]: TEST-55-OOMD-slowrule.service: Main process exited, \
code=exited, status=2/INVALIDARGUMENT
systemd[1]: TEST-55-OOMD-slowrule.service: Failed with result \
'exit-code'.
Try to detect at the beginning of the test and skip the test case if it
happens.
di is allocated while iterating over p.partitions. If the array is
empty,
the loop is skipped and di remains NULL, but the code still assigns
image
metadata through it.
Return EBADMSG before accessing di in this case.
Found by Linux Verification Center (linuxtesting.org) with Svace.
When adding new groups, putgrent_with_members() is called with the newly
constructed group n, but the error path logs gr->gr_name. gr belongs to
the
earlier loop over the existing group file and may be NULL after EOF.
Log n.gr_name instead.
Found by Linux Verification Center (linuxtesting.org) with Svace.
SUSE does not provide a vmlinux.h so the package is not built with
CO-RE support, hence the test fails. This was previously masked
by the fact that python3-packaging was never installed, so the
test always skipped everywhere as it could not detect the kernel
version.
Follow-up for c310106c15
Each <type>_id is picked from a Unit.List '{}' call and immediately
queried via a second Unit.List '{"name":"<id>"}' call. When the picked
unit is tied to a user session (run-user-N.mount, user@N.service,
user-runtime-dir@N.service, session-cN.scope) it may be torn down
between the two calls if the user manager happens to exit at that
moment, causing a spurious NoSuchUnit failure.
Filter those out of the candidate lists for mount/service/scope.
Excerpt from a failed CI run showing the race:
[ 2631.395826] systemd[1]: user-runtime-dir@4711.service: Trying to enqueue job user-runtime-dir@4711.service/stop/fail
[ 2632.101414] systemd[1]: user-runtime-dir@4711.service: Got final SIGCHLD for state stop.
[ 2632.182537] TEST-74-AUX-UTILS.sh[3051]: + mount_id=run-user-4711.mount
[ 2632.186760] TEST-74-AUX-UTILS.sh[3099]: + varlinkctl call /run/systemd/io.systemd.Manager io.systemd.Unit.List '{"name":"run-user-4711.mount"}'
[ 2632.189752] systemd[1]: varlink-api-3-3: Sending message: {"error":"io.systemd.Unit.NoSuchUnit"}
[ 2632.190752] TEST-74-AUX-UTILS.sh[3099]: Method call io.systemd.Unit.List() failed: io.systemd.Unit.NoSuchUnit
[ 2632.191594] TEST-74-AUX-UTILS.sh[101]: Subtest /usr/lib/systemd/tests/testdata/units/TEST-74-AUX-UTILS.varlinkctl-unit.sh failed
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
The attempt to fix SIGILL in TEST-55-OOMD actually made the test more
flaky, as the chosen stress-ng method generates less pressure, so it
often fails.
Try a different approach: try to detect that SIGILL caused the unit
calling stress-ng to fail, and skip gracefully in that case.
The numbers of signals vary by arch. On the common arches the signals
listed here all use the same numbers, but people are likely to use
this on more fringe architectures too, so let's use symbolic names
instead.
Also the comment about gdb "hitting the same kill" didn't make sense.
The syntax is a bit baroque, but using a helper variable does not work.
Also shellcheck complains about $[ ] which would have made this more
legible.
The attempt to fix SIGILL in TEST-55-OOMD actually made the test more
flaky, as the chosen stress-ng method generates less pressure, so it
often fails.
Try a different approach: try to detect that SIGILL caused the unit
calling stress-ng to fail, and skip gracefully in that case.
This allows multi-call binaries to be easily invoked with a different
name. After installation, the name is set by creating a symlink. But in
build directories, we don't create the symlinks. (There are also other
ways to achieve the same thing, e.g. zsh supports $ARGV0, and exec -a
can be used, but those are either non-portable or are more complicated
to use.) The primary use-case for me is to test --help output for
multicall binaries.
Also reorder the help for env vars to group the more generic ones near
the top.
This was initially proposed in https://github.com/systemd/systemd/pull/24054,
but there were some comments about the implementation. I had a branch
with the patch, but I don't think I ever actually submitted it as a
pull request.
When building with Meson >=1.3.0, use the native `fs.relative_to()`
helper instead of shelling out to `realpath`.
This is more portable: BusyBox `realpath` does not support the
`--relative-to` option.
nspawn started logging at notice level when exiting a container:
root@sid-arm64:~# exit
logout
Container sid-arm64 exited successfully.
Failed to unregister machine in system context, ignoring: No such device or address
Looks like it's racing with machined which already noticed it was
gone and remove it. Downgrade to debug on ENXIO.
Follow-up for 1e084aad7d
On some architectures like m68k, alignof(void*) is 2, not sizeof(void*)
(which is 4). So the natural alignment of struct Option is 2 and
sizeof(Option) == 26.
However, each variable placed in the SYSTEMD_OPTIONS ELF section via
_OPTION() carries _alignptr_ (= __attribute__((aligned(sizeof(void*))))),
which forces each entry to start at a 4-byte boundary. The linker
therefore inserts 2 bytes of padding between adjacent entries, producing
an actual stride of 28 in the section.
option_parse() iterates over the section with pointer arithmetic
("opt++"), which advances by sizeof(Option) == 26 and lands inside the
padding. The fields read back as zero, and since commit cf88903637
("tree-wide: get rid of most uses of ALIGN_PTR") added the ordering
assertion below, the resulting "0 < 0" trips it when running --help:
Assertion 'opt->id < (opt + 1)->id' failed at src/shared/options.c:116, function option_parse(). Aborting.
#0 0xc0a7b248 in ?? () from /usr/lib/m68k-linux-gnu/libc.so.6
#1 0xc0a7b2ce in pthread_kill () from /usr/lib/m68k-linux-gnu/libc.so.6
#2 0xc0a2edc6 in raise () from /usr/lib/m68k-linux-gnu/libc.so.6
#3 0xc0a1c128 in abort () from /usr/lib/m68k-linux-gnu/libc.so.6
#4 0xc05c6f78 in option_parse (options=0x0, options_end=0x0, state=0xc09ca968) at ../src/shared/options.c:116
Fix this by applying _alignptr_ to the struct definition itself, so that
sizeof(Option) is padded up to a multiple of sizeof(void*) and matches
the actual on-disk stride. Add an assert_cc() so any future regression
is caught at compile time.
The same latent bug applies to Verb and TestFunc, which use the same
section-placement pattern. Their natural sizeof was already a multiple
of sizeof(void*) so no crash was observed, but apply the same fix
defensively.
Follow-up for cf88903637
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Not the case on the sparc architecture. Replace with two translation
functions. The poll to epoll variant isn't used yet but will be used
when we add io-uring support to sd-event so we already add it here so
that it is in place.
Follow-up for cf4c65afa8
Luca Boccassi
Zbigniew Jędrzejewski-Szmek
Yu Watanabe
Frantisek Sumsal
Mikhail Nogin
Daan De Meyer
naly zzwd
Ali Ciloglu
Fco. Javier F. Serrador
Lennart Poettering
vad