The hash tables are only ever set up in newly created journal files, in
space that was just allocated via posix_fallocate() and hence reads as
zeroes anyway, so zeroing the items explicitly is unnecessary.
It is also harmful: the sequential write faults of the memzero() make the
kernel's mmap readahead instantiate the table in ever larger folios, up to
the size of the readahead window, on file systems that support large folios
in the page cache (e.g. XFS, and ext4 since kernel 6.16). And since write
faults on shared file mappings dirty the whole folio, each later update of
a single 16 byte hash bucket then has the whole folio written back. Without
the explicit zeroing, the table's pages are only faulted in when buckets
are first accessed, in random order, and hence mostly as order-0 folios.
Measured on Linux 6.18 with a new journal file (SystemMaxFileSize=50M),
appending 15 syslog-like entries with 23 fields each per second for 120s,
while writing back the file's dirty pages every second, as happens on
machines under page cache pressure, where reclaim keeps waking up the
flusher threads. Bytes written to the (loop) device, median and range of
4-5 runs:
before after
ext4 182.8M (178.4M-185.7M) 41.2M (31.6M-41.9M)
xfs 172.7M (161.0M-182.4M) 40.1M (39.7M-44.9M)
This also avoids writing out the whole data hash table once for every new
journal file, i.e. ~2.8% of the maximum file size.
Related: #40262
(cherry picked from commit 6db7cf38d40e8fe6a11392d2f0cc044ed5acf812)
An AF_UNIX stream receive can combine a message without file descriptors
with a following SCM_RIGHTS-bearing message. This associates the received
descriptors with the earlier message even when the sender uses separate
sendmsg() calls.
Use the existing boundary-limited read path whenever FD input is enabled
on stream sockets, not only for protocol upgrades. Include the buffered
delimiter suffix in the peek search so a split CRLF delimiter cannot
over-read into the next message either.
Record the input socket type when attaching the descriptors so the new
FD-triggered limit does not truncate packet sockets. Explicit bounded
reads retain their existing semantics.
Add regressions for a oneway request followed by an FD-bearing request,
a continued reply followed by an FD-bearing terminal reply, a split CRLF
delimiter followed by an FD-bearing message, and batched messages on
SOCK_SEQPACKET and SOCK_DGRAM.
Fixes#43823
(cherry picked from commit 3fe57d03a3)
Previously, only sd_nfnl_nft_message_new_setelems() takes the 'add'
flag. Let's all functions take the boolean parameter for consistency.
(cherry picked from commit 534a25c885)
This updates a couple of links talking about Varlink:
1. anything that refers to specific Varlink concepts now point to the
UAPI.20 spec
2. More general references to Varlink as a whole point to both the
varlink.org domain and the spec.
When a multi-byte delimiter straddles two reads, json_stream_parse()
only scans the bytes from the latest read. As a result, a QMP reply
whose "\r\n" terminator is split between reads is not recognized.
Include up to delimiter_size - 1 trailing bytes from the existing buffer
in the next scan, so delimiters spanning reads can be matched.
Add regression coverage for a split CRLF delimiter followed by another
message in the same read. Check that both objects are parsed correctly
and that no buffered input remains afterwards.
Vacuuming unlinks and deallocates archived journal files. A reader
may have already selected an entry from such a file, so final
validation can fail after candidate selection succeeded.
If the selected file is now unlinked, remove it and retry candidate
selection. Preserve errors from linked files, including the EIO
reported after SIGBUS recovery from a truncated mapping.
The last five hand-written help() functions are replaced by COMMAND()
descriptions: test-homectl-prompts, test-libudev,
test-modem-manager-mock, test-chase-manual, and test-journal-append
(which open-coded the help inside the option parsing loop). None of the
binaries are installed, so no man pages are referenced and they are not
added to the introspection smoke test.
Negative return values, e.g. from option parsing, were propagated verbatim
as the exit status of main() and truncated to meaningless values (-EINVAL
became 234). The positive-failure variant is needed because the binary may
exit with EXIT_TEST_SKIP.
Also propagate EXIT_TEST_SKIP instead of suppressing it.
As mentioned in a05fe6684a, glibc provides two
wrappers for epoll_pwait2() on 32-bit architectures, and sys/epoll.h,
together with features-time64.h, selects the appropriate one.
To avoid depending on symbols introduced in newer glibc releases, that
commit relied on the glibc internal macro __USE_TIME_BITS64 to determine
which wrapper to use. However, that is incorrect on some architectures.
See:
https://sourceware.org/git/?p=glibc.git;a=commit;h=a4ed0471d71739928a0d0fa3258b3ff3b158e9b9
The above glibc commit introduces a new internal macro,
__USE_TIME64_REDIRECTS, which correctly describes whether the time64
wrapper should be used. However, we cannot rely on it because it was
introduced in glibc 2.40, while our baseline remains glibc 2.34.
Moreover, even after raising the baseline, relying on glibc internal
implementation details is undesirable.
Instead, call the epoll_pwait2 syscall directly. This avoids any
dependency on glibc wrapper selection logic while preserving the original
goal of avoiding newer glibc symbols.
As a side effect, this also enables epoll_pwait2() on musl, which does not
currently provide a libc wrapper for it.
Currently, epoll_pwait2() is only used by sd-event, so this is also the
simplest solution.
I had a situation where Dell Wyse 3040 with SK Hynix eMMC stopped booting
after upgrading from Fedora 42 to Fedora 44 with systemd 259.8.
The card reports a product name containing byte 0x92, but the kernel event
also contains the valid module alias needed to load mmc_block:
MMC_NAME=H8G4a\222
MODALIAS=mmc:block
I inserted the mmc_block kernel module manually and booted successfully.
Then after enabling udevd debug logging and retriggering the MMC add event
via
udevadm control -l debug
udevadm trigger -v -c add -s mmc
udevadm settle
I saw the failure in the "journalctl -b -u systemd-udevd.service"
output:
mmc0:0001: sd-device: Refusing invalid property: MMC_NAME=H8G4a\222
mmc0:0001: sd-device: Failed to add property 'MMC_NAME=H8G4a<92>': Invalid argument
sd-device-monitor(manager): Failed to create device from received message: Invalid argument
Since PR 41001, strict property validation returns EINVAL for the bad
MMC_NAME. device_new_from_nulstr() propagated that error and discarded
the whole event, so udev never processed MODALIAS and did not load
mmc_block.
Skip ordinary properties whose keys or values fail validation. Do not store
them, but continue processing the remaining properties. Required and typed
properties are handled separately and remain fatal when invalid, as do
malformed records and all other errors.
Closes: https://github.com/systemd/systemd/issues/42930
Related: https://github.com/systemd/systemd/issues/41339
Related: https://github.com/systemd/systemd/issues/43008
Similar: https://github.com/systemd/systemd/pull/42316
Similar: https://github.com/systemd/systemd/pull/43061
Regression: https://github.com/systemd/systemd/pull/41001
The connect callback is called after sd_varlink_server_add_connection()
succeeds. Do not call the disconnect callback for connections that have
not been established yet.
Previously, sd_varlink_server_add_connection_pair() did not decrement
the connection counters when json_stream_attach_fds() failed.
Set the ucred before calling json_stream_attach_fds() so that
sd_varlink_close() also decrements the counters when it fails.
This makes json_stream_attach_fds() and json_stream_connect_fd_pair()
leave the passed file descriptors with the caller on failure. Callers
should call TAKE_FD() only after a successful call.
Also make qmp_client_connect_fd() always take ownership of the passed
file descriptor, including on failure. Previously, ownership of the
passed file descriptor on failure was unclear and depended on where
the function failed.
Only trailing empty components have special meaning:
they request appending the built-in defaults.
Share the validation between normal path lookup and
systemd-analyze verify, so verify does not compose
values that path lookup later rejects.
Propagate generator path parse errors instead of
turning them into empty generator paths.
Follow-up for: cf7d80a5fe
compare_boot_ids() (from 262299dccb) skips
the tail entry realtime timestamp ordering when the compared locations
are from two different machines (and hence have two different machine
IDs) - in that case we fall through to a realtime comparison between the
two locations. This works fine except for journals that were imported
via systemd-journal-remote - the machine ID in the journal header is set
to the receiving machine's ID instead of the original one; but the
original machine ID is kept in the _MACHINE_ID= field.
This, however, defeats the check in compare_boot_ids() so we treat the
remote journals as local ones and try to compare the boots based on
their tail entry realtime timestamps, which results in grouping the
entries by their boot ID instead of interleaving them by their realtime
timestamp.
Given that we (transitively) call journal_file_read_tail_timestamp()
from compare_boot_ids(), where we both (re)set the newest_machine_id
field to the machine ID from the journal header and have the tail entry
at hand, let's try to get the _MACHINE_ID= field from the tail entry and
set newest_machine_id to that if it's present and valid. This way
compare_boot_ids() will correctly determine that the entries are from
two different machines and compare_locations() eventually falls through
to the entry realtime check, so we end up with properly interleaved
entries instead of entries grouped by boot ID.
The performance impact should be pretty negligible in vast majority of
cases, as we'd do this only when:
- opening a new file
- comparing entries across different boots
- advancing to a next location
The first case is a one-time thing only per journal. The second and
third case might call this a bit more frequently, but we fall through to
the _MACHINE_ID= parsing only if the journal file got a new entry since
the last check. The most significant impact would be on journals that
are being actively written into, but even then iterating through the
handful of typical fields is negligible compared to what the rest of the
function does. Also, since 8f8ab4bf98 the
journal_file_read_tail_timestamp() calls are ratelimited when walking
through the journal, lessening the impact even further.
Resolves: #34169
Since 262299dccb compare_locations() falls
through to a realtime-based comparison in compare_boot_ids() if the
compared entries have different boot IDs (and seqnum IDs). This can,
however, cause us to pick up the wrong entry/file on systems with
unreliable/missing RTC, where early-boot entries of a boot might have
earlier realtime timestamp compared to the previous boot.
For example, here's a minimal reproducer consisting of two journal files
containing three entries from two separate boot IDs:
$ journalctl --file one.journal -o export
__CURSOR=s=ba37c62fcacf406aad088166c4963cd1;i=1;b=c220efc91dd440deb89d82c98ddb11a5;m=186a0;t=1e8480;x=39d4323a8e5a248d
__REALTIME_TIMESTAMP=2000000
__MONOTONIC_TIMESTAMP=100000
__SEQNUM=1
__SEQNUM_ID=ba37c62fcacf406aad088166c4963cd1
_BOOT_ID=c220efc91dd440deb89d82c98ddb11a5
NUMBER=1
$ journalctl --file two.journal -o export
__CURSOR=s=9d15197efcc54d8ca5d6e7cce28373e2;i=1;b=686dc35e78f64b6ba126018ab6e8bae9;m=186a0;t=f4240;x=57db1c7bd38b37ed
__REALTIME_TIMESTAMP=1000000
__MONOTONIC_TIMESTAMP=100000
__SEQNUM=1
__SEQNUM_ID=9d15197efcc54d8ca5d6e7cce28373e2
_BOOT_ID=686dc35e78f64b6ba126018ab6e8bae9
NUMBER=2
__CURSOR=s=9d15197efcc54d8ca5d6e7cce28373e2;i=2;b=686dc35e78f64b6ba126018ab6e8bae9;m=30d40;t=2dc6c0;x=3513732159948113
__REALTIME_TIMESTAMP=3000000
__MONOTONIC_TIMESTAMP=200000
__SEQNUM=2
__SEQNUM_ID=9d15197efcc54d8ca5d6e7cce28373e2
_BOOT_ID=686dc35e78f64b6ba126018ab6e8bae9
NUMBER=3
The first file (one.journal) contains a single entry from the first boot
(c220efc91dd440deb89d82c98ddb11a5) and a realtime timestamp of 2s. The
second file (two.journal) contains two entries from the second boot
(686dc35e78f64b6ba126018ab6e8bae9) - the first entry has a realtime
timestamp of 1s (i.e. _earlier_ than the tail timestamp of the first
boot), and the second entry has a realtime timestamp of 3s (i.e. after
a time correction).
This works fine for forward iteration, but seeking directly to the
second entry is when things go south:
First entry:
$ python3 <<EOF
c = "s=ba37c62fcacf406aad088166c4963cd1;i=1;b=c220efc91dd440deb89d82c98ddb11a5;m=186a0;t=1e8480;x=39d4323a8e5a248d"
from systemd import journal; j = journal.Reader(path="."); j.seek_cursor(c); j.get_next(); print(j.test_cursor(c))
EOF
True
Second entry:
$ python3 <<EOF
c = "s=9d15197efcc54d8ca5d6e7cce28373e2;i=1;b=686dc35e78f64b6ba126018ab6e8bae9;m=186a0;t=f4240;x=57db1c7bd38b37ed"
from systemd import journal; j = journal.Reader(path="."); j.seek_cursor(c); j.get_next(); print(j.test_cursor(c))
EOF
False
That's because:
- we call sd_journal_seek_cursor(...) and parse it (seqnum, seqnum_id,
boot_id, realtime, ...)
- we call sd_journal_next() -> real_journal_next() which iterates over
all open journal files to find all candidate entries and pick the
best one
- for each journal file we call next_beyond_location()
- for next_beyond_location(one.journal) we get:
- last_direction is _DIRECTION_INVALID, so we call find_location_with_matches()
- we fall all the way through to journal_file_move_to_entry_by_realtime()
- here we try to find the first entry with realtime >= 1s; this file
has only one entry with realtime = 2s which matches
- we got a match and new_file is unset, so we set it to one.journal
- for next_beyond_location(two.journal) we get:
- we fall through to find_location_with_matches()
- here we have a matching seqnum, so we call journal_file_move_to_entry_by_seqnum()
- we got another match but new_file is set, so we call compare_locations(two.journal, one.journal):
- we fall through to compare_boot_ids()
- we get the tail timestamps of both journal files and return
CMP(second boot, first boot) -> CMP(3s, 2s) = 1
- this bubbles up back to real_journal_next() where found = false, so
new_file remains one.journal (the wrong journal file)
- this is followed by set_location(one.journal), which makes the
following sd_journal_test_cursor() call fail, as the
boot_id/seqnum_id/... of the current position don't match with the
cursor
To fix this, let's track the journal file that holds the exact cursor
match, and override the chosen candidate if needed once we process all
open journals.
Resolves: #31516
Commit 52ceba53d3 made multipart replies append in receive
order, but each append walked the chain from the head.
Cache the tail on the chain head, so new fragments can be
linked directly. The visible next-pointer chain and order stay
unchanged.
Follow-up for: 52ceba53d3
sd_bus_message carries an array of header field offsets and a counter,
but their only reader was part of the D-Bus v2/GVariant sealing path
removed in 0dd4876815 ("sd-bus: drop D-Bus version 2 format support").
There are three remaining callers of message_extend_fields(), for
SD_BUS_MESSAGE_HEADER_DESTINATION, _PATH, and _INTERFACE, and all of
them pass false for add_offset. The receive path does not populate the
array either, so nothing reads or writes it any more.
Let's drop the two fields and the now unused add_offset argument and
branch.
This reduces sizeof(sd_bus_message) from 792...
$ gdb -batch build-baseline/test-bus-benchmark \
-ex 'ptype /o struct sd_bus_message'
...
/* 688 | 8 */ usec_t timeout;
/* 696 | 80 */ size_t header_offsets[10];
/* 776 | 4 */ unsigned int n_header_offsets;
/* XXX 4-byte hole */
/* 784 | 8 */ uint64_t read_counter;
/* total size (bytes): 792 */
to 704 bytes:
$ gdb -batch build-patched/test-bus-benchmark \
-ex 'ptype /o struct sd_bus_message'
...
/* 688 | 8 */ usec_t timeout;
/* 696 | 8 */ uint64_t read_counter;
/* total size (bytes): 704 */
On Fedora 43 aarch64 with glibc 2.42, the usable allocation for a
method-call message falls from 808 to 728 bytes, a reduction of 9.9%.
In my tests, retaining 400,000 method-call messages reduces in median
peak RSS from 363M to 332M.
Using `test-bus-benchmark chart direct 500ms` across message sizes from
1 byte to 2 MiB one can also see things are around 1% faster, which is
another nice incidental boost.
Accept RFC4122 UUID URN strings with the `urn:uuid:` prefix in
sd_id128_from_string(), while keeping plain 128-bit IDs and regular
UUID strings working as before.
This partially reverts 13ce62caba.
Sometimes, the kernel passes non-safe characters in uevent files,
such as the UNIQ= property for USB serial devices.
Instead of rejecting the entire uevent file, let's only ignore the
specific key-value pairs that contain non-safe characters.
Fixes#43008.
Let's take inspiration from string_is_safe() and take a flags field that
allows fine tuning the validation.
Then port over all current users of either function to the new logic.
Note that this *does* change behaviour in various cases:
1. Generally: we'll now always accept the full UAPI.10 alphabet,
including the "~" and "^" characters. As far as I can see there's no
downside to this liberalization as none of the current consumers of
the two functions uses these characters for anything else.
2. systemd-analyze compare-version will now accept version strings with
"_" and "+" without complaining. I see no downside here, it just
normalizes these debugging tools, to make them accept what most our
other tools accept.
3. "bootctl link" will not accept empty version strings anymore
Which is a bugfix I guess.
4. vpick will now refuse "_" and "+" in version strings. It kinda
already did, because when parsing versions from filenames it uses "_"
and "+" as name, architecture and attempt counter separators. We now
systematically refuse it everywhere else in vpick too. This is hence
a clean-up.
Fixes: #28906
Replaces: #42815#41937
When no prefix is specified, portablectl inspect first tries the
prefix derived from the image name. This keeps inspect aligned with
attach behavior.
If that lookup finds no matching units, retry with validated
PORTABLE_PREFIXES read from the image os-release. This makes inspect
work for images whose filename does not match their portable service
prefix.
Keep metadata error handling explicit so request-construction failures
are not logged twice, while sd_bus_call() failures still include the
inspect context.
Add a TEST-29-PORTABLE regression case for a directory image whose
name does not match the portable service prefix.
Fixes#37296.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
This are to the existing sd_varlink_call_and_upgrade() what
sd_varlink_callb() and sd_varlink_callbo() are to sd_varlink_call():
they put together an object on the fly, via the usual JSON builder
logic.
Previously, even though sd-journal does not support bzip2 and gzip
compression, the dlopen ELF notes for those libraries were still being
attached to libsystemd.so and various journal-handling executables.
This occurred because the unified compression/decompression interfaces
handled all formats unconditionally, causing the compiler and linker to
pull in all associated dlopen notes across the board.
To resolve this, split these functions into generic and journal-specific
variants (e.g., introducing compress_blob_journal() and decompress_blob_journal()).
The journal-specific variants only handle formats actually supported by
the journal (LZ4, XZ, and ZSTD).
By updating sd-journal, journald, journalctl, and related utilities to
use these new `_journal` interfaces and switching them to the narrower
COMPRESS_JOURNAL_NOTE macro, we ensure that unnecessary dlopen notes
(for bzip2 and gzip) are no longer embedded into these binaries.
When a rate limit is armed, for example via StartLimitInterval=, and the
machine is suspended, the rate limit's "clock" is suspended as well,
since the elapse checks use CLOCK_MONOTONIC. This then causes unexpected
situations where a rate limit armed via StartLimitInterval=1h, followed
by a 10 hour suspend, would elapse after 11 hours total.
Let's avoid this by switching the rate limits to CLOCK_BOOTTIME, which
works the same as CLOCK_MONOTONIC, but accounts for the time spent in
suspend as well.
There's one slight concern when it comes to upgrade path - the old
"begin" value of the rate limit is stored as CLOCK_MONOTONIC, but after
upgrading systemd and serializing/deserializing the state it will be
suddenly compared against now(CLOCK_BOOTTIME), which might cause some
rate limits to elapse "prematurely". But this is just a one-time thing,
after which the rate limit timers should re-assess themselves.
Resolves: #42912
add_object_vtable_internal() inserts BusVTableMember entries into the
global vtable_methods/vtable_properties sets as it walks the vtable, but
only sets node_vtable.node (which gates the disconnect-time cleanup)
after the loop. A failure mid-loop thus left the already-inserted
members behind, dangling once the slot's interface string and the node's
path were freed, so a later registration reading those keys hit a
use-after-free. Remove this slot's members on the fail path.
The test asserts the failed registration leaves no member behind.
Follow-up for 19befb2d5f
The CAN_HASH branch of bus_match_run() that iterates the array values of
an argNhas= match dispatches a callback per value but, unlike the leaf
and manual-iteration branches, does not re-check bus->match_callbacks_modified
afterwards. A callback that adds or removes a match (freeing the node)
would then have the next hashmap_get(node->compare.children, ...) read
freed memory. Add the same check the sibling branches already have.
Follow-up for 198b158f49
netlink_slot_allocate() links the slot into nl->slots and takes an nl
reference, but sd_netlink_call_async() and netlink_add_match_internal()
hold it in a _cleanup_free_, so an error after allocation frees the slot
with a bare free()
Follow-up for ee38400bba
SecretLUL
Totoro
Yu Watanabe
Chris Down
Lennart Poettering
Michael Renner
Zbigniew Jędrzejewski-Szmek
Jakub Stasiak
ShiroKSH
dongshengyuan
Frantisek Sumsal
Eric Boucher
Luca Boccassi