sd_journal_next() sweeps every open journal file to find the next entry, and
next_beyond_location() rate-limits its tail timestamp refresh per file. Since
ratelimit_below() reads the clock itself, iterating a journal spread over a
hundred files meant a hundred clock_gettime() calls for every single entry,
nearly all of them only to conclude that the one second interval had not
elapsed yet. On a 730M journal in 100 files this was the single largest cost in
journalctl, over a fifth of the run.
Add ratelimit_below_at(), which takes the timestamp from the caller, and read
the clock once per sweep in real_journal_next() instead. ratelimit_below()
keeps its behaviour and becomes a wrapper.
The timestamp is now sampled once for the whole sweep rather than once per
file. The interval being rate-limited is a second and a sweep takes
microseconds, so this makes no practical difference.
On that journal, journalctl -o export goes from 4.02s to 2.93s, and the default
journalctl -o short from 3.30s to 2.14s.
This decoded every character twice: utf8_encoded_valid_unichar() decodes
internally to do its validation and then throws the codepoint away, and the
caller immediately called utf8_encoded_to_unichar() again to get the very same
codepoint back.
Fix that where it originates rather than at the call site: add
utf8_encoded_valid_unichar_full(), which optionally returns the character it had
to decode anyway, and make utf8_encoded_valid_unichar() a wrapper that passes
NULL. Callers that want both the validity and the value now decode once.
On top of that add a bulk fast path, since almost everything we are asked about
is plain printable ASCII: test eight bytes at a time for that, i.e. that no byte
is >= 0x80, none is < 0x20 and none is 0x7f. A remainder of less than eight
bytes is copied into a word padded with spaces so that it can be tested the same
way. A word that does not qualify falls through to the per character path, so
tabs, newlines and multibyte sequences are handled exactly as before.
The function sits on journalctl's per-field output path, so this is measurable:
on a 730M journal spread over 100 files, journalctl -o export goes from 4.02s to
about 2.9s.
Also extend the tests, which so far covered neither DEL nor the C1 range nor
allow_newline=false: place each of the interesting bytes at each offset within
the eight byte word, and check a multibyte character straddling the end of one.
The new code was verified to agree with the old one on 41945090 inputs, for both
values of allow_newline: every byte string of length zero to three over all 256
byte values, four million random strings over uniform bytes, over an ASCII-heavy
alphabet and over an alphabet of encoding boundary bytes, and a corpus placing
every byte value at every offset of buffers of every length from eight to
twenty-four bytes, with valid and truncated multibyte sequences straddling the
word boundary.
This is split out of #43058, but makes sense on its own: and makes
strip_tab_ansi() a lore more complete and correct.
ptybroker is going to use for the ansi sequence parser for other
purposes too, but making strip_tab_ansi() work better is enough of a
reason to have this I think.
It's nice to use --attach since it validates our images. So far, it
strictly insisted on checking that the image was a DDI however, which
limits its use for certain usecases.
Add "--relax" to address this limitation.
Fixes: #43605
systemd has two UTF-8 decoders, and the console width path uses the
permissive one.
`utf8_encoded_to_unichar()` performs no overlong check, no
`unichar_is_valid()` check, and still accepts the obsolete five and six
byte forms. Those checks live in the wrapper
`utf8_encoded_valid_unichar()`, which is what `utf8_is_valid()` and
`ellipsize_mem()` use. `utf8_char_console_width()` called the bare
decoder directly.
Checking both against `utf8_is_valid_n()` over all 4,244,897,280 NUL
free byte strings of length one to four, which is the entire UTF-8
encoding space:
| decoder | disagreements with `utf8_is_valid_n()` |
| --- | --- |
| `utf8_encoded_valid_unichar()` | 0 |
| `utf8_encoded_to_unichar()` | 8,835,838, always more permissive |
So `utf8_console_width()` returned a finite width for strings
`utf8_is_valid()` rejects. `format-table.c` measures a cell with
`utf8_console_width()` and, when it does not fit, hands it to
`ellipsize()`, which validates strictly and returns NULL. That is
reported as `-ENOMEM`:
```
lead surrogate valid_utf8=0 utf8_console_width=25 ellipsize=NULL
-> table_format FAILS: Cannot allocate memory
```
A surrogate, overlong, noncharacter or five byte sequence in any cell
failed the whole table with a bogus out of memory error, and only once
that column was narrow enough to need truncation, so whether it happened
depended on the width of the user's terminal. At other widths the
invalid sequence was printed to the terminal instead. Invalid cells are
now refused consistently with `-EINVAL`, which is what the sequences
`utf8_encoded_to_unichar()` already rejected always produced.
The second commit fixes a related disagreement. `ellipsize_mem()` and
`align_string_mem()` open coded character width as `unichar_iswide(c) ?
2 : 1`, while `unichar_console_width()` counts a tab as eight cells.
`format-table.c` uses both: it finds a field too wide with
`utf8_console_width()`, then asks `ellipsize()` to fit that many
columns, and `ellipsize()` thought a tab was one cell, so the result
still overflowed the column that the measurement had just asked it to
fit. A table pinned to `table_set_width(20)` emitted lines of 25, 44 and
39 columns. This uses `unichar_console_width()`, which was split out for
exactly this purpose in 5c0d4b5061.
Both commits add tests that fail before the change and pass after. The
full unit suite is otherwise unaffected.
Adds EncryptAddToken=<type>[:<source>:<value>] to repart.d partition
files, registering one or more LUKS2 token objects in the partition
header after the keyslot is created. This allows custom token plugins
(e.g. a TDX KBS plugin) to be wired up at provisioning time without a
separate cryptsetup token import step.
The type name is always explicit and extra token fields can be supplied as
a JSON file (:file:/path) or base64-encoded JSON (:data:<base64>). The
type and keyslots fields are always injected by repart. The option is
repeatable; an empty value clears all tokens for the partition.
Supported with Encrypt=key-file, tpm2, and key-file+tpm2.
If KExecsCount > 0 then do not attempt to mark a UKI, as the stub (and also
any addon and the bootloader) will not have been exercised, so there's no
guaranteed a full reboot with the same image will actually work.
Fixes https://github.com/systemd/systemd/issues/43738
A triple backtick inside prose made the fence count odd, so a body with an
inline marker plus a valid suggestion block got cut at the block's closing
fence and left it open. Only fences at the start of a line delimit blocks,
so count and strip those alone.
This level of debugging might be over the top, so I kept this commit
separate. If reviewers think that it's too much, let's drop it.
It turns out that SELinux defeats getenv_secure(), so the expected
variables are not read, c.f.
https://bugzilla.redhat.com/show_bug.cgi?id=2533000#c8.
Reviews keep hitting the 50 minute timeout. The coordinator agent that was
supposed to launch reviewers, watch the clock, stop stragglers, validate and
write the result does none of that reliably: reviewers get stopped and lose
their findings, foreground validators can't be stopped, deadlines get
overshot by minutes, and one leftover agent keeps the process alive until
the job fails. Piling more rules into its prompt hasn't helped.
Do the orchestration in the shell instead. Run one bounded claude invocation
per review lens in parallel under timeout(1), each returning its findings as
structured output, merge them with jq and run one bounded validator over the
candidates. A lens that times out is just reported as failed. The post job
builds the tracking comment from what it actually posted, so the model no
longer authors the summary or the review bookkeeping. Agents run from an
empty directory with all inputs outside it, so the sandbox keeps both the
checkout and the inputs read-only.
Let's make sure that we have wider regression coverage:
1. Verify duplicate, empty, binary, and repeated fields are stored and
enumerated correctly
2. Exercise reader position invalidation after seeking and changing
matches
3. Check cursor, data, unique-field, and field-enumeration lifetime
semantics
4. Test journal opening and following across invalid files, file
descriptor ownership, renames, unlinks, polling, immutable files, forked
readers, etc, etc
When systemd-loop@.service is instantiated for a block device (e.g. a
CD-ROM drive, which is the primary usecase now that --relax is used),
%f resolves to the backing block device rather than to the loopback
block device that was attached on top of it. In that case ExecStop=
hence invoked --detach on a block device that is not a loopback device
at all, which made loop_device_set_autoclear() a silent no-op, leaking
the attachment.
Hence, if --detach is passed a block device that is not a loopback
block device itself, search for the loopback block device backed by it,
the same way we already do for regular files. The kernel reports the
backing inode for block device nodes too, so the same inode matching
logic applies.
Our primary usecase for this service is to deal with sector issues
around CDROMs. For that insisting on proper DDI layout and a root
partition is overkill. Let's hence relax the logic for this.
Fixes: #43605
It's nice to use --attach since it validates our images. So far, it
strictly insisted on checking that the image was a DDI however, which
limits its use for certain usecases.
Add "--relax" to address this limitation.
The set of file based output types is open-coded in three places (and a
fourth one is about to be added), let's define it once, next to the
other ExecInput/ExecOutput predicates.
strv_rebreak_lines() was the last place left that decoded with the permissive
utf8_encoded_to_unichar() and then asked unichar_console_width() how wide the
result is, so it accepted and measured overlong encodings, surrogates,
noncharacters and the obsolete five and six byte forms that utf8_console_width()
now refuses.
That matters because callers mix the two within a single width computation.
sd-varlink-idl.c sizes its indent with utf8_console_width() and then breaks the
body with strv_rebreak_lines(); verbs.c does the same. One half of the sum
refused a sequence that the other half happily assigned a width to.
Use utf8_char_console_width(), which does the validation and the width in one
step, and advance with utf8_next_char() rather than the decoded length. The
function already refused invalid UTF-8, so the only change is which sequences
count as invalid; it now matches utf8_is_valid(), as everywhere else.
ellipsize_mem() and align_string_mem() open coded the width of a character as
unichar_iswide(c) ? 2 : 1, but unichar_console_width() counts a tab as eight
cells. The two disagreed, and format-table.c uses both: it decides a field is too
wide with utf8_console_width(), then asks ellipsize() to fit it into that many
columns, and ellipsize() believed a tab was one cell. The ellipsized cell then
still overflowed the column, measured by the very function that had just asked
for it to be truncated. A table pinned to table_set_width(20) emitted lines of
25, 44 and 39 columns. align_string_mem() undercounted the same way and so padded
already overlong cells further.
Use unichar_console_width(), split out for this purpose in 5c0d4b5061. While at
it, let align_string_mem() decode with utf8_char_console_width() as well, so that
the second pass shares both the width model and the validity check with the
measuring pass, instead of keeping a more permissive decoder of its own around
for the next caller to trip over. Its "count invalid chars as 1" fallback now
skips the whole rejected sequence rather than one byte, so that the continuation
bytes are not charged a cell each of their own; the branch should be unreachable
anyway, since our caller only gets there after utf8_console_width() succeeded on
the same string.
Note that unichar_console_width() charges a tab a flat eight cells, while a
terminal advances to the next multiple of eight, so this is deliberately an upper
bound: the two agree only when the tab itself starts on a multiple of eight. A
per character helper cannot do better, because a tab's width is a function of the
column it starts in, and neither ellipsize_mem() nor align_string_mem() is told
what column its string starts at. Overestimating is the safe direction: it can
leave a column padded short, but it cannot overflow one.
ascii_ellipsize_mem() equates bytes with character cells, which holds for ASCII
except for the tab, so keep tabs off that fast path. Like the ANSI sequence case
just above it, such strings now take the generic path, which means they get the
Unicode ellipsis and one reserved cell rather than "..." and three, even in a non
UTF-8 locale.
utf8_char_console_width() decoded with utf8_encoded_to_unichar(), which does not
check for overlong encodings, does not check unichar_is_valid(), and still
accepts the obsolete five and six byte forms. Those checks live in the wrapper
utf8_encoded_valid_unichar(), which is what utf8_is_valid() and ellipsize_mem()
use. Over all 4244897280 NUL free byte strings of length one to four, that is the
whole UTF-8 encoding space, the wrapper agrees with utf8_is_valid_n() everywhere,
while the bare decoder disagrees on 8835838 of them, always by accepting more.
So utf8_console_width() assigned a finite width to strings that utf8_is_valid()
refuses. format-table.c measures a cell with utf8_console_width() and, if it does
not fit, hands it to ellipsize(), which validates strictly, returns NULL, and is
reported as -ENOMEM. A surrogate, overlong, noncharacter or five byte sequence in
any cell therefore failed the whole table with a bogus out of memory error, and
only once the column happened to be narrow enough to require truncation, so
whether it happened at all depended on the width of the user's terminal. At other
widths the invalid sequence was printed to the terminal instead.
Decode with utf8_encoded_valid_unichar() so that how wide we think a string is
agrees with whether we consider it valid UTF-8 in the first place. Invalid cells
are now refused consistently with -EINVAL, the way the sequences that
utf8_encoded_to_unichar() already rejected always were.
Note that for tables this makes the refusal total rather than per field:
table_data_requested_width_height() turns a SIZE_MAX width into -EINVAL, which
table_format() propagates, so a cell in one of the newly rejected encodings now
fails the whole table at every width. systemd-dissect and repart --dry-run put
GPT partition names read straight off the image into cells without filtering
them, so a crafted partition name suppresses the whole listing where before it
cost one field, and only at some terminal widths. That is the intended trade:
the alternative is to keep emitting raw invalid bytes to the terminal at
whichever widths happen not to need truncation. Should degrading the cell rather
than the table turn out to be preferable, the place for it is where the text
enters the table, via utf8_escape_invalid() and friends, not in a helper whose
job is to say how wide a string is.
This also makes the assert(w >= 0) in utf8_escape_non_printable_full() hold by
construction rather than by the accident that its own validation was stricter
than the width helper's.
So far strip_tab_ansi() was very superficial: it only focussed on OSC
and CSI sequences, and left the other ANSI sequences in touch.
Let's make it more powerful and port it to the more complete
AnsiSeqParser logic, and strip *all* ANSI sequences.
For the relevant usecases this seems like the better, safer choice,
since it's good to strip too much rather than too little, when
stripping ANSI sequences.
So far we didn't limit the number of jumps when decompressing a DNS
packet. This can lead to some extra work with specially crafted packets
that send a staircase of thousands of descending compression pointers,
which can then be amplified by referencing such staircase from thousands
of names.
Given that legitimate DNS packets need only a handful of jumps in the
compression chains, let's limit the number of jumps per a single name to
256 (the same limit as, for example, Unbound currently has).
Reported by opensec-intelligence (OpenSec).
namespace_enter() tried to become root after joining a user namespace
but ignored failures, so if root wasn't mapped we'd just carry on with
our original UID/GID. Let's make that an error instead.
Dropping supplementary groups stays best effort, as setgroups() might
be denied (e.g. by the nsresourced BPF LSM) even when root is mapped.
Some callers do need to enter namespaces without root, so add
NAMESPACE_ENTER_KEEP_UID_GID_IF_ROOT_UNMAPPED for those. It's used by
systemd-analyze unit-shell and by the credential refresh in the user
manager. Everything else enters containers or namespaces we allocated
ourselves, where root is always mapped.
Introspection is implemented for everything that has a command-line
parser, not only for select binaries.
Standarize on "rate-limit" as the spelling in docs.
Various other smaller tweaks and clarifications.
This is especially necessary since GCC 16.x.
=====
$ gcc --version
gcc (GCC) 16.2.1 20260819 (Red Hat 16.2.1-2)
Copyright (C) 2026 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
=====
Without this change, /usr/include/limits.h is not included, hence POSIX
constants are not defined:
=====
$ echo '#include <limits.h>' | gcc -E -H -isystem ~/git/systemd/src/include/musl - 2>&1 >/dev/null | grep '^\..*limit'
. /home/watanabe/git/systemd/src/include/musl/limits.h
.. /usr/lib/gcc/x86_64-redhat-linux/16/include/limits.h
... /usr/lib/gcc/x86_64-redhat-linux/16/include/syslimits.h
=====
With this change, /usr/include/limits.h is included, thus POSIX
constants are defined:
=====
$ echo '#include <limits.h>' | gcc -E -H -isystem ~/git/systemd/src/include/musl - 2>&1 >/dev/null | grep '^\..*limit'
. /home/watanabe/git/systemd/src/include/musl/limits.h
.. /usr/lib/gcc/x86_64-redhat-linux/16/include/limits.h
... /usr/include/limits.h
...... /usr/include/linux/limits.h
.. /usr/lib/gcc/x86_64-redhat-linux/16/include/limits.h
=====
Replaces #43824.
The _const_ attribute can be only used for functions.
Fixes the following warnings on Fedora rawhide:
```
../src/core/namespace.h:260:75: warning: '__const__' attribute only applies to functions [-Wignored-attributes]
260 | char* bpf_delegate_to_string(uint64_t u, const char * (*parser)(uint64_t) _const_);
| ^
../src/basic/../fundamental/macro.h:80:32: note: expanded from macro '_const_'
80 | #define _const_ __attribute__((__const__))
| ^
In file included from ../src/core/unit.c:24:
../src/shared/condition.h:87:63: warning: '__const__' attribute only applies to functions [-Wignored-attributes]
87 | typedef const char* (*condition_to_string_t)(ConditionType t) _const_;
| ^
../src/basic/../fundamental/macro.h:80:32: note: expanded from macro '_const_'
80 | #define _const_ __attribute__((__const__))
| ^
```
Only PCI was treated as a "less specific" bus that should not be
assigned if USB was already assigned, so non-PCI USB controllers
(under platform/DT or ACPI) would have all their devices *not*
treated as USB at all, resulting in external peripherals being
treated as internal ones on DT platforms.
Clarify these assignments into specificity/nesting groups instead
of having some arbitrary "X must be before Y" comments.
Fixes a4381cae8b.
We generally want that the exposed strings for enum values and the
internal names for them stay in sync. Let's make sure this is the case
for the ConfidentialComputing enum.
This just changes the internal names, so that they match the external
ones. No user visible change, just some search and replace.
Armaan Sandhu
Yu Watanabe
Roman Perehonchuk
Nandakumar Raghavan
Luca Boccassi
Daan De Meyer
Zbigniew Jędrzejewski-Szmek
Georg Müller
Lennart Poettering
Frantisek Sumsal
Chris Down
Val Packett
Lennart Poettering
zouge666