The last five hand-written help() functions are replaced by COMMAND()
descriptions: test-homectl-prompts, test-libudev,
test-modem-manager-mock, test-chase-manual, and test-journal-append
(which open-coded the help inside the option parsing loop). None of the
binaries are installed, so no man pages are referenced and they are not
added to the introspection smoke test.
Negative return values, e.g. from option parsing, were propagated verbatim
as the exit status of main() and truncated to meaningless values (-EINVAL
became 234). The positive-failure variant is needed because the binary may
exit with EXIT_TEST_SKIP.
Also propagate EXIT_TEST_SKIP instead of suppressing it.
As mentioned in a05fe6684a, glibc provides two
wrappers for epoll_pwait2() on 32-bit architectures, and sys/epoll.h,
together with features-time64.h, selects the appropriate one.
To avoid depending on symbols introduced in newer glibc releases, that
commit relied on the glibc internal macro __USE_TIME_BITS64 to determine
which wrapper to use. However, that is incorrect on some architectures.
See:
https://sourceware.org/git/?p=glibc.git;a=commit;h=a4ed0471d71739928a0d0fa3258b3ff3b158e9b9
The above glibc commit introduces a new internal macro,
__USE_TIME64_REDIRECTS, which correctly describes whether the time64
wrapper should be used. However, we cannot rely on it because it was
introduced in glibc 2.40, while our baseline remains glibc 2.34.
Moreover, even after raising the baseline, relying on glibc internal
implementation details is undesirable.
Instead, call the epoll_pwait2 syscall directly. This avoids any
dependency on glibc wrapper selection logic while preserving the original
goal of avoiding newer glibc symbols.
As a side effect, this also enables epoll_pwait2() on musl, which does not
currently provide a libc wrapper for it.
Currently, epoll_pwait2() is only used by sd-event, so this is also the
simplest solution.
I had a situation where Dell Wyse 3040 with SK Hynix eMMC stopped booting
after upgrading from Fedora 42 to Fedora 44 with systemd 259.8.
The card reports a product name containing byte 0x92, but the kernel event
also contains the valid module alias needed to load mmc_block:
MMC_NAME=H8G4a\222
MODALIAS=mmc:block
I inserted the mmc_block kernel module manually and booted successfully.
Then after enabling udevd debug logging and retriggering the MMC add event
via
udevadm control -l debug
udevadm trigger -v -c add -s mmc
udevadm settle
I saw the failure in the "journalctl -b -u systemd-udevd.service"
output:
mmc0:0001: sd-device: Refusing invalid property: MMC_NAME=H8G4a\222
mmc0:0001: sd-device: Failed to add property 'MMC_NAME=H8G4a<92>': Invalid argument
sd-device-monitor(manager): Failed to create device from received message: Invalid argument
Since PR 41001, strict property validation returns EINVAL for the bad
MMC_NAME. device_new_from_nulstr() propagated that error and discarded
the whole event, so udev never processed MODALIAS and did not load
mmc_block.
Skip ordinary properties whose keys or values fail validation. Do not store
them, but continue processing the remaining properties. Required and typed
properties are handled separately and remain fatal when invalid, as do
malformed records and all other errors.
Closes: https://github.com/systemd/systemd/issues/42930
Related: https://github.com/systemd/systemd/issues/41339
Related: https://github.com/systemd/systemd/issues/43008
Similar: https://github.com/systemd/systemd/pull/42316
Similar: https://github.com/systemd/systemd/pull/43061
Regression: https://github.com/systemd/systemd/pull/41001
The connect callback is called after sd_varlink_server_add_connection()
succeeds. Do not call the disconnect callback for connections that have
not been established yet.
Previously, sd_varlink_server_add_connection_pair() did not decrement
the connection counters when json_stream_attach_fds() failed.
Set the ucred before calling json_stream_attach_fds() so that
sd_varlink_close() also decrements the counters when it fails.
This makes json_stream_attach_fds() and json_stream_connect_fd_pair()
leave the passed file descriptors with the caller on failure. Callers
should call TAKE_FD() only after a successful call.
Also make qmp_client_connect_fd() always take ownership of the passed
file descriptor, including on failure. Previously, ownership of the
passed file descriptor on failure was unclear and depended on where
the function failed.
Only trailing empty components have special meaning:
they request appending the built-in defaults.
Share the validation between normal path lookup and
systemd-analyze verify, so verify does not compose
values that path lookup later rejects.
Propagate generator path parse errors instead of
turning them into empty generator paths.
Follow-up for: cf7d80a5fe
compare_boot_ids() (from 262299dccb) skips
the tail entry realtime timestamp ordering when the compared locations
are from two different machines (and hence have two different machine
IDs) - in that case we fall through to a realtime comparison between the
two locations. This works fine except for journals that were imported
via systemd-journal-remote - the machine ID in the journal header is set
to the receiving machine's ID instead of the original one; but the
original machine ID is kept in the _MACHINE_ID= field.
This, however, defeats the check in compare_boot_ids() so we treat the
remote journals as local ones and try to compare the boots based on
their tail entry realtime timestamps, which results in grouping the
entries by their boot ID instead of interleaving them by their realtime
timestamp.
Given that we (transitively) call journal_file_read_tail_timestamp()
from compare_boot_ids(), where we both (re)set the newest_machine_id
field to the machine ID from the journal header and have the tail entry
at hand, let's try to get the _MACHINE_ID= field from the tail entry and
set newest_machine_id to that if it's present and valid. This way
compare_boot_ids() will correctly determine that the entries are from
two different machines and compare_locations() eventually falls through
to the entry realtime check, so we end up with properly interleaved
entries instead of entries grouped by boot ID.
The performance impact should be pretty negligible in vast majority of
cases, as we'd do this only when:
- opening a new file
- comparing entries across different boots
- advancing to a next location
The first case is a one-time thing only per journal. The second and
third case might call this a bit more frequently, but we fall through to
the _MACHINE_ID= parsing only if the journal file got a new entry since
the last check. The most significant impact would be on journals that
are being actively written into, but even then iterating through the
handful of typical fields is negligible compared to what the rest of the
function does. Also, since 8f8ab4bf98 the
journal_file_read_tail_timestamp() calls are ratelimited when walking
through the journal, lessening the impact even further.
Resolves: #34169
Since 262299dccb compare_locations() falls
through to a realtime-based comparison in compare_boot_ids() if the
compared entries have different boot IDs (and seqnum IDs). This can,
however, cause us to pick up the wrong entry/file on systems with
unreliable/missing RTC, where early-boot entries of a boot might have
earlier realtime timestamp compared to the previous boot.
For example, here's a minimal reproducer consisting of two journal files
containing three entries from two separate boot IDs:
$ journalctl --file one.journal -o export
__CURSOR=s=ba37c62fcacf406aad088166c4963cd1;i=1;b=c220efc91dd440deb89d82c98ddb11a5;m=186a0;t=1e8480;x=39d4323a8e5a248d
__REALTIME_TIMESTAMP=2000000
__MONOTONIC_TIMESTAMP=100000
__SEQNUM=1
__SEQNUM_ID=ba37c62fcacf406aad088166c4963cd1
_BOOT_ID=c220efc91dd440deb89d82c98ddb11a5
NUMBER=1
$ journalctl --file two.journal -o export
__CURSOR=s=9d15197efcc54d8ca5d6e7cce28373e2;i=1;b=686dc35e78f64b6ba126018ab6e8bae9;m=186a0;t=f4240;x=57db1c7bd38b37ed
__REALTIME_TIMESTAMP=1000000
__MONOTONIC_TIMESTAMP=100000
__SEQNUM=1
__SEQNUM_ID=9d15197efcc54d8ca5d6e7cce28373e2
_BOOT_ID=686dc35e78f64b6ba126018ab6e8bae9
NUMBER=2
__CURSOR=s=9d15197efcc54d8ca5d6e7cce28373e2;i=2;b=686dc35e78f64b6ba126018ab6e8bae9;m=30d40;t=2dc6c0;x=3513732159948113
__REALTIME_TIMESTAMP=3000000
__MONOTONIC_TIMESTAMP=200000
__SEQNUM=2
__SEQNUM_ID=9d15197efcc54d8ca5d6e7cce28373e2
_BOOT_ID=686dc35e78f64b6ba126018ab6e8bae9
NUMBER=3
The first file (one.journal) contains a single entry from the first boot
(c220efc91dd440deb89d82c98ddb11a5) and a realtime timestamp of 2s. The
second file (two.journal) contains two entries from the second boot
(686dc35e78f64b6ba126018ab6e8bae9) - the first entry has a realtime
timestamp of 1s (i.e. _earlier_ than the tail timestamp of the first
boot), and the second entry has a realtime timestamp of 3s (i.e. after
a time correction).
This works fine for forward iteration, but seeking directly to the
second entry is when things go south:
First entry:
$ python3 <<EOF
c = "s=ba37c62fcacf406aad088166c4963cd1;i=1;b=c220efc91dd440deb89d82c98ddb11a5;m=186a0;t=1e8480;x=39d4323a8e5a248d"
from systemd import journal; j = journal.Reader(path="."); j.seek_cursor(c); j.get_next(); print(j.test_cursor(c))
EOF
True
Second entry:
$ python3 <<EOF
c = "s=9d15197efcc54d8ca5d6e7cce28373e2;i=1;b=686dc35e78f64b6ba126018ab6e8bae9;m=186a0;t=f4240;x=57db1c7bd38b37ed"
from systemd import journal; j = journal.Reader(path="."); j.seek_cursor(c); j.get_next(); print(j.test_cursor(c))
EOF
False
That's because:
- we call sd_journal_seek_cursor(...) and parse it (seqnum, seqnum_id,
boot_id, realtime, ...)
- we call sd_journal_next() -> real_journal_next() which iterates over
all open journal files to find all candidate entries and pick the
best one
- for each journal file we call next_beyond_location()
- for next_beyond_location(one.journal) we get:
- last_direction is _DIRECTION_INVALID, so we call find_location_with_matches()
- we fall all the way through to journal_file_move_to_entry_by_realtime()
- here we try to find the first entry with realtime >= 1s; this file
has only one entry with realtime = 2s which matches
- we got a match and new_file is unset, so we set it to one.journal
- for next_beyond_location(two.journal) we get:
- we fall through to find_location_with_matches()
- here we have a matching seqnum, so we call journal_file_move_to_entry_by_seqnum()
- we got another match but new_file is set, so we call compare_locations(two.journal, one.journal):
- we fall through to compare_boot_ids()
- we get the tail timestamps of both journal files and return
CMP(second boot, first boot) -> CMP(3s, 2s) = 1
- this bubbles up back to real_journal_next() where found = false, so
new_file remains one.journal (the wrong journal file)
- this is followed by set_location(one.journal), which makes the
following sd_journal_test_cursor() call fail, as the
boot_id/seqnum_id/... of the current position don't match with the
cursor
To fix this, let's track the journal file that holds the exact cursor
match, and override the chosen candidate if needed once we process all
open journals.
Resolves: #31516
Commit 52ceba53d3 made multipart replies append in receive
order, but each append walked the chain from the head.
Cache the tail on the chain head, so new fragments can be
linked directly. The visible next-pointer chain and order stay
unchanged.
Follow-up for: 52ceba53d3
sd_bus_message carries an array of header field offsets and a counter,
but their only reader was part of the D-Bus v2/GVariant sealing path
removed in 0dd4876815 ("sd-bus: drop D-Bus version 2 format support").
There are three remaining callers of message_extend_fields(), for
SD_BUS_MESSAGE_HEADER_DESTINATION, _PATH, and _INTERFACE, and all of
them pass false for add_offset. The receive path does not populate the
array either, so nothing reads or writes it any more.
Let's drop the two fields and the now unused add_offset argument and
branch.
This reduces sizeof(sd_bus_message) from 792...
$ gdb -batch build-baseline/test-bus-benchmark \
-ex 'ptype /o struct sd_bus_message'
...
/* 688 | 8 */ usec_t timeout;
/* 696 | 80 */ size_t header_offsets[10];
/* 776 | 4 */ unsigned int n_header_offsets;
/* XXX 4-byte hole */
/* 784 | 8 */ uint64_t read_counter;
/* total size (bytes): 792 */
to 704 bytes:
$ gdb -batch build-patched/test-bus-benchmark \
-ex 'ptype /o struct sd_bus_message'
...
/* 688 | 8 */ usec_t timeout;
/* 696 | 8 */ uint64_t read_counter;
/* total size (bytes): 704 */
On Fedora 43 aarch64 with glibc 2.42, the usable allocation for a
method-call message falls from 808 to 728 bytes, a reduction of 9.9%.
In my tests, retaining 400,000 method-call messages reduces in median
peak RSS from 363M to 332M.
Using `test-bus-benchmark chart direct 500ms` across message sizes from
1 byte to 2 MiB one can also see things are around 1% faster, which is
another nice incidental boost.
Accept RFC4122 UUID URN strings with the `urn:uuid:` prefix in
sd_id128_from_string(), while keeping plain 128-bit IDs and regular
UUID strings working as before.
This partially reverts 13ce62caba.
Sometimes, the kernel passes non-safe characters in uevent files,
such as the UNIQ= property for USB serial devices.
Instead of rejecting the entire uevent file, let's only ignore the
specific key-value pairs that contain non-safe characters.
Fixes#43008.
Let's take inspiration from string_is_safe() and take a flags field that
allows fine tuning the validation.
Then port over all current users of either function to the new logic.
Note that this *does* change behaviour in various cases:
1. Generally: we'll now always accept the full UAPI.10 alphabet,
including the "~" and "^" characters. As far as I can see there's no
downside to this liberalization as none of the current consumers of
the two functions uses these characters for anything else.
2. systemd-analyze compare-version will now accept version strings with
"_" and "+" without complaining. I see no downside here, it just
normalizes these debugging tools, to make them accept what most our
other tools accept.
3. "bootctl link" will not accept empty version strings anymore
Which is a bugfix I guess.
4. vpick will now refuse "_" and "+" in version strings. It kinda
already did, because when parsing versions from filenames it uses "_"
and "+" as name, architecture and attempt counter separators. We now
systematically refuse it everywhere else in vpick too. This is hence
a clean-up.
Fixes: #28906
Replaces: #42815#41937
When no prefix is specified, portablectl inspect first tries the
prefix derived from the image name. This keeps inspect aligned with
attach behavior.
If that lookup finds no matching units, retry with validated
PORTABLE_PREFIXES read from the image os-release. This makes inspect
work for images whose filename does not match their portable service
prefix.
Keep metadata error handling explicit so request-construction failures
are not logged twice, while sd_bus_call() failures still include the
inspect context.
Add a TEST-29-PORTABLE regression case for a directory image whose
name does not match the portable service prefix.
Fixes#37296.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
This are to the existing sd_varlink_call_and_upgrade() what
sd_varlink_callb() and sd_varlink_callbo() are to sd_varlink_call():
they put together an object on the fly, via the usual JSON builder
logic.
Previously, even though sd-journal does not support bzip2 and gzip
compression, the dlopen ELF notes for those libraries were still being
attached to libsystemd.so and various journal-handling executables.
This occurred because the unified compression/decompression interfaces
handled all formats unconditionally, causing the compiler and linker to
pull in all associated dlopen notes across the board.
To resolve this, split these functions into generic and journal-specific
variants (e.g., introducing compress_blob_journal() and decompress_blob_journal()).
The journal-specific variants only handle formats actually supported by
the journal (LZ4, XZ, and ZSTD).
By updating sd-journal, journald, journalctl, and related utilities to
use these new `_journal` interfaces and switching them to the narrower
COMPRESS_JOURNAL_NOTE macro, we ensure that unnecessary dlopen notes
(for bzip2 and gzip) are no longer embedded into these binaries.
When a rate limit is armed, for example via StartLimitInterval=, and the
machine is suspended, the rate limit's "clock" is suspended as well,
since the elapse checks use CLOCK_MONOTONIC. This then causes unexpected
situations where a rate limit armed via StartLimitInterval=1h, followed
by a 10 hour suspend, would elapse after 11 hours total.
Let's avoid this by switching the rate limits to CLOCK_BOOTTIME, which
works the same as CLOCK_MONOTONIC, but accounts for the time spent in
suspend as well.
There's one slight concern when it comes to upgrade path - the old
"begin" value of the rate limit is stored as CLOCK_MONOTONIC, but after
upgrading systemd and serializing/deserializing the state it will be
suddenly compared against now(CLOCK_BOOTTIME), which might cause some
rate limits to elapse "prematurely". But this is just a one-time thing,
after which the rate limit timers should re-assess themselves.
Resolves: #42912
add_object_vtable_internal() inserts BusVTableMember entries into the
global vtable_methods/vtable_properties sets as it walks the vtable, but
only sets node_vtable.node (which gates the disconnect-time cleanup)
after the loop. A failure mid-loop thus left the already-inserted
members behind, dangling once the slot's interface string and the node's
path were freed, so a later registration reading those keys hit a
use-after-free. Remove this slot's members on the fail path.
The test asserts the failed registration leaves no member behind.
Follow-up for 19befb2d5f
The CAN_HASH branch of bus_match_run() that iterates the array values of
an argNhas= match dispatches a callback per value but, unlike the leaf
and manual-iteration branches, does not re-check bus->match_callbacks_modified
afterwards. A callback that adds or removes a match (freeing the node)
would then have the next hashmap_get(node->compare.children, ...) read
freed memory. Add the same check the sibling branches already have.
Follow-up for 198b158f49
netlink_slot_allocate() links the slot into nl->slots and takes an nl
reference, but sd_netlink_call_async() and netlink_add_match_internal()
hold it in a _cleanup_free_, so an error after allocation frees the slot
with a bare free()
Follow-up for ee38400bba
The tag-match loop emits 6 instructions per tag into a fixed 512-entry
stack array without a bounds check (unlike the subsystem loop below),
so enough tags overflow the stack. Refuse with -E2BIG instead.
Follow-up for b1c097af8d
Let's not manually create comments, but use the usual
varlink_idl_format_comment() function which does line break handling
properly.
This polishes the output for very narrow outputs, since we'll line break
these synthetic comments too.
This makes each `foo_cflags` dependency define a `SYSTEMD_CFLAGS_MARKER_FOO`
macro, and checks if the macro is set when headers provided by external
libraries are included.
With this, we can fail fast at compile time if necessary `_cflags`
dependencies are omitted in meson.build. Missing dependencies found by this
mechanism have been added across the tree.
The idea is that we can build a container by building a single-binary
systemd:
```console
$ meson setup build-static --default-library=static --prefer-static --auto-features=disabled -Dbuild-static=true -Dsystemd-multicall-binary=true && ninja -C build-static systemd
$ mkdir /var/tmp/container/usr/lib -p
$ cp build-static/systemd /var/tmp/container/usr/lib/
$ echo 'ID=quick' >/var/tmp/container/usr/lib/os-release
$ systemd-nspawn --restrict-address-families=af_unix --register=no --private-users=managed -D /var/tmp/container/ /usr/lib/systemd
░ Spawning container container on /var/tmp/container.
░ Press Ctrl-] three times within 1s to kill container; two times followed by r
░ to reboot container; two times followed by p to poweroff container.
Selected user namespace base 1855193088 and range 65536.
systemd 262~devel running in system mode (-PAM -AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK -SECCOMP -GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP -LIBARCHIVE)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Detected first boot.
Welcome to Linux!
Initializing machine ID from container UUID.
Failed to open netlink, ignoring: Address family not supported by protocol
Applying preset policy.
Populated /etc with preset unit settings.
Unit default.target not found.
Falling back to graphical.target.
Mount unit not supported, skipping *MountsFor= dependencies.
Queued start job for default target graphical.target.
[ OK ] Reached target sysinit.target.
[ OK ] Reached target basic.target.
System is tainted: unmerged-bin:var-run-bad
[ OK ] Reached target multi-user.target.
[ OK ] Reached target graphical.target.
Startup finished in 61ms.
```
The container can be reloaded with SIGTERM, powered off with SIGRTMIN+4,
etc. SIGRTMIN+5 should cause a reboot but it currently fails:
```
...
Rebooting.
Container container is being rebooted.
Failed to attach root directory: Invalid argument
Failed to receive mount namespace fd from outer child: Input/output error
```
It's a bug … somewhere, but probably not caused by the linking changes
being done here.
Some state files were prefixed with the machine name which protected
against collisions when running concurrent machines in a systemd unit
with RuntimeDirectory= set. However, the socket files tpm.sock and
control were not, which caused a race. Prefixing these socket files
does not work because the path gets too long when both the subdirectory
and the socket file within it get unique identifiers.
Instead of prefixing all files, we can rather always create a
subdirectory and use simple names within in. This makes paths shorter
in the normal case and protects against races with RuntimeDirectory=
were instead of directly reusing RUNTIME_DIRECTORY we also create the
normal per-machine subdirectory. Since this here is about runtime state
it should not impact any running VMs/containers. The ssh-proxy looks
only at the normal case and does not support RUNTIME_DIRECTORY, so no
impact there as well.
When we try to build with -static, the linker refuses dlopen and
various functions related to nss. This commit adds the basic support
to compile with -static and allow the link to pass.
Add more shutdown timestamps, export them via D-Bus/Varlink, use them in
`systemd-analyze time`, and preserve them across kexec via LUO. This is
useful to measure reboot performance with added measurement points and
more granular intervals.
journal_file_read_tail_timestamp() is called unconditionally in
next_beyond_location() for every file on every iteration step,
resulting in O(N x files) volatile mmap reads. For large queries
like 'journalctl -n 1000000' this makes the command unusably slow
(~5 minutes on systems with many journal files).
Rather than suppressing the call entirely (which would make the
inotify path fully load-bearing for cross-boot ordering), rate-limit
it to at most once per second per file. This reduces the overhead
from O(N x files) to O(T x files) where T is the iteration time in
seconds, while still providing periodic refresh as a fallback for
any missed inotify events and keeping cross-boot ordering
reasonably fresh.
Embed a RateLimit struct in JournalFile for this purpose.
Measured improvement on a real system: 5:24 -> 2:39 (-51%) for
'journalctl -n 1000000'.
Fixes: #42521
Regression introduced in: #26355 (34af74946e)
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Zbigniew Jędrzejewski-Szmek
Yu Watanabe
Jakub Stasiak
ShiroKSH
dongshengyuan
Frantisek Sumsal
Chris Down
Eric Boucher
Lennart Poettering
Luca Boccassi
Kai Lüke