json_stream_format_queue() drains queued output items into the output buffer and
stages their fds in n_output_fds, relying on the downstream sendmsg() to deliver
bytes-and-ancillary atomically as one SCM_RIGHTS message. If the output buffer
already holds bytes (from a prior fast-path enqueue that hasn't been sent yet or
from a partial write), concatenating a new fd-bearing item's JSON into it means
the next sendmsg() ships the combined bytes with those fds attached — violating
the per-message fd boundary on transports where that boundary is load-bearing.
Bail out of the drain loop when we would cross that boundary, so the next
write() first sends the buffered bytes with no ancillary, then pulls the
fd-bearing item into a clean buffer and ships it on its own sendmsg.
This only produces an observable difference for SOCK_SEQPACKET / SOCK_DGRAM
consumers: on those transports each sendmsg() is its own datagram with its own
SCM_RIGHTS cmsg, so whether we concatenate matters. On AF_UNIX SOCK_STREAM
(today's sole consumer shape, used by sd-varlink and the QMP client) the kernel
absorbs a preceding non-scm skb forward into the next scm-bearing skb's recv,
so per-sendmsg separation is invisible to the receiver anyway — the guard is
cheap defensive sender hygiene there, not a behaviour change. It becomes load-
bearing the moment a SEQPACKET/DGRAM consumer wires JsonStream up.
Add --forward-journal=FILE|DIR to forward the container's journal
entries to the host via systemd-journal-remote. When specified,
nspawn starts systemd-journal-remote listening on a Unix socket,
bind-mounts it into the container at /run/host/journal/socket, and
passes a journal.forward_to_socket credential pointing to it.
Add --forward-journal-max-use=, --forward-journal-keep-free=,
--forward-journal-max-file-size=, and --forward-journal-max-files=
to configure disk usage limits for the forwarded journal.
Consolidate nspawn's per-machine on-disk state under a single runtime
directory at /run/systemd/nspawn/<machine>/. The container rootdir
mount point moves from /tmp/nspawn-root-XXXXXX to <runtime_dir>/root,
the unix-export directory moves from
/run/systemd/nspawn/unix-export/<machine> to <runtime_dir>/unix-export,
and the journal-remote socket lives at
<runtime_dir>/journal-remote-socket. Update ssh-generator and
ssh-proxy to follow the new unix-export path layout.
Extract fork_journal_remote() into fork-notify.{c,h} as a shared
helper used by both nspawn and vmspawn, replacing vmspawn's
start_systemd_journal_remote().
Extract runtime_directory_make() into path-lookup.{c,h} as a shared
helper used by both nspawn and vmspawn, replacing vmspawn's inline
runtime directory creation logic.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Every dlopen_xxx() helper now takes an int log_level argument. It is
passed through to dlopen_many_sym_or_warn() (which in turn propagates it
to dlopen_verbose() for the library-not-installed case), and is used by
the fallback stub when support for the library is not compiled in to
emit a "<lib> support is not compiled in." message at the caller's level.
Callers pass LOG_DEBUG when gracefully degrading, or a higher level when
the failure should surface, and no longer need to log redundantly at the
call site.
As part of this, dlopen_bpf_full() (which already took a log_level) is
merged into dlopen_bpf() rather than keeping both.
The static inline fallbacks used to live in the headers, which required
pulling log.h in from every header that declared a dlopen_xxx(). Move
them into the .c files instead: the declaration is always outside the
#if HAVE_XXX block, the impl sits outside the outer #if HAVE_XXX wrap
with its own internal #if HAVE_XXX/#else/#endif, and apparmor-util.c,
idn-util.c, libmount-util.c and pam-util.c are now always compiled so
they can host their stubs.
The help string is adjusted/reworded. In particular, [a, b) is used
as notation to show a closed-open range, instead of the unusual <a; b).
--help is shown in --help.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Split the single directory fd that chaseat() used to take into two separate
fds: a root_fd that sets the chroot boundary (symlinks may not escape it,
absolute symlinks resolve relative to it), and a dir_fd that path resolution
starts from. This makes the chroot semantics of chaseat() explicit at every
call site instead of encoding them in the CHASE_AT_RESOLVE_IN_ROOT flag,
which is removed. It also decouples the starting directory from the root
boundary, so callers can descend from any inode inside the tree without
having to reopen the root separately.
XAT_FDROOT passed as root_fd means "no containment" (host root); as dir_fd
it means "start at root_fd". For a smoother transition, AT_FDCWD is also
accepted as root_fd and treated as XAT_FDROOT. When root_fd points to a
directory that is actually the host root, it is normalized to XAT_FDROOT
up front so the existing shortcut path can kick in.
Absolute paths returned by chaseat() are now relative to root_fd, and
relative paths are relative to dir_fd. The result is absolute only when
there is no chroot boundary (root_fd is XAT_FDROOT), or when an absolute
symlink made resolution jump out of the dir_fd subtree; otherwise callers
get a relative path they can feed straight back into an openat()-style
call against dir_fd. Specifically, when dir_fd == root_fd and we're not
operating on the host's root directory, we return a relative path even if
we received an absolute path or resolved an absolute symlink to allow
passing the path directly to openat() style functions. We do this to not
have to go modify every caller of chaseat() to make sure they deal properly
with any absolute paths they might receive. Only when root_fd != dir_fd do
we have to return an absolute path to indicate that the path is relative to
root_fd and not dir_fd.
The shortcut that skips the per-component walk is reworked around a new
chase_xopenat() helper that funnels CHASE_NOFOLLOW, CHASE_MUST_BE_* and
CHASE_TRIGGER_AUTOFS through xopenat_full()'s O_NOFOLLOW, O_DIRECTORY,
XO_REGULAR, XO_SOCKET and XO_TRIGGER_AUTOMOUNT flags. As a result these
flags no longer force us off the shortcut and can be dropped from
CHASE_NO_SHORTCUT_MASK, and the old openat_opath_with_automount() helper
goes away. A CHASE_MUST_BE_ANY alias is introduced for shortcut callers
(stat/access paths) that don't go through xopenat_full() and still need
to bail on those flags locally.
All *_and_* helpers built on top of chaseat() (chase_and_openat,
chase_and_opendirat, chase_and_statat, chase_and_accessat,
chase_and_fopenat_unlocked, chase_and_unlinkat, chase_and_open_parent_at)
gain the same root_fd parameter, and every call site in the tree is
ported to the new signature.
chase_and_open() is also fixed to correctly handle CHASE_EXTRACT_FILENAME
without CHASE_PARENT.
The json-stream API previously exposed JsonStreamQueueItem and several
functions operating on it (json_stream_make_queue_item(),
json_stream_enqueue_item(), json_stream_queue_item_free(),
json_stream_queue_item_get_data()). These existed solely to support
sd-varlink's "defer-and-modify" pattern for streaming replies, where a
reply is held back so its "continues" field can be set before
transmission. This is a varlink protocol concern that should not leak
into the generic transport layer.
Similarly, the fd pushing API (json_stream_push_fd(),
json_stream_reset_pushed_fds()) and the pushed_fds state lived inside
JsonStream, even though fd-to-message association is a protocol-level
concern managed entirely by sd-varlink.
Rework the API so that:
- JsonStreamQueueItem and all its functions become static to
json-stream.c. The only output API is now json_stream_enqueue_full()
(accepting explicit fds) and the inline json_stream_enqueue() wrapper
for the common no-fds case.
- The pushed_fds state moves from JsonStream into sd_varlink, where
sd_varlink_push_fd() and sd_varlink_reset_fds() manage it directly.
- The deferred reply in sd-varlink changes from a JsonStreamQueueItem*
to a plain sd_json_variant* plus a separate previous_fds/n_previous_fds
pair, keeping the protocol-specific bookkeeping in sd-varlink where it
belongs.
- A new varlink_enqueue() helper wraps json_stream_enqueue_full() with
the varlink connection's pushed fds, transferring fd ownership to the
queue item on success.
qmp-client.c is fixed to use the new API as well.
sd_varlink_error() deliberately returns a negative errno mapped from
the error id on success so callbacks can `return sd_varlink_error(...);`
to enqueue the reply and propagate a matching errno at once. When
varlink_dispatch_method() dispatches a configured error sentinel itself,
it doesn't need that mapping — but it was treating any negative return
as a dispatch failure and logging "Failed to process sentinel" even
though the error reply had been successfully enqueued.
Detect success via the state transition to VARLINK_PROCESSED_METHOD
instead, so only genuine enqueue failures are logged.
Move json_stream_description(), json_stream_log(), and
json_stream_log_errno() from json-stream.c into json-stream.h so that
consumers like the QMP client can use the same description-prefixed
logging that json-stream itself uses internally.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
I promised in https://github.com/systemd/systemd/pull/41426 that its the
final update for coccinelle pointer deref checks. However it turned out
there is this coccinelle/parsing_hacks.h that I wasn't aware of. The
file missed some important things like _cleanup_(x) that prevented
coccinelle to check a bunch of functions.
This PR adds some missing defines to the parsing_hacks.h and fixes the
missing asserts(). I apologize that its a bit long (and frankly boring)
and that I missed this earlier.
The last commit contains one small behavior change (ret in
sd_varlink_idl_parse() is now really optional) but the big one is very
mechanical.
We already have checks in place during compression that limit the data
we compress, so they shouldn't decompress to anything larger than
DATA_SIZE_MAX unless they've been tampered with. Let's make this
explicit and limit all our decompress_blob() calls in journal-handling
code to that limit.
One possible scenario this should prevent is when one tries to open and
verify a journal file that contains a compression bomb in its payload:
```
$ ls -lh test.journal
-rw-rw-r--+ 1 fsumsal fsumsal 1.2M Apr 12 15:07 test.journal
$ systemd-run --user --wait --pipe -- build-local/journalctl --verify --file=$PWD/test.journal
Running as unit: run-p682422-i4875779.service
000110: Invalid hash (00000000 vs. 11e4948d73bdafdd)
000110: Invalid object contents: Bad message
File corruption detected at /home/fsumsal/repos/@systemd/systemd/test.journal:272 (of 1249896 bytes, 0%).
FAIL: /home/fsumsal/repos/@systemd/systemd/test.journal (Bad message)
Finished with result: exit-code
Main processes terminated with: code=exited, status=1/FAILURE
Service runtime: 48.051s
CPU time consumed: 47.941s
Memory peak: 8G (swap: 0B)
```
Same could be, in theory, possible with just `journalctl --file=`, but
the reproducer would be a bit more complicated (haven't tried it, yet).
Lastly, the change in journal-remote is mostly hardening, as the maximum
input size to decompress_blob() there is mandated by MHD's connection
memory limit (set to JOURNAL_SERVER_MEMORY_MAX which is 128 KiB at the
time of writing), so the possible output size there is already quite
limited (e.g. ~800 - 900 MiB for xz-compressed data).
Convert string fields to varlink enums in io.systemd.Unit
Following
https://github.com/systemd/systemd/pull/39391#discussion_r2489599449,
convert all configuration setting fields in the io.systemd.Unit varlink
interface from bare SD_VARLINK_STRING to proper enum types, adding type
safety to the IDL.
This converts ~30 fields across ExecContext, CGroupContext, and
UnitContext, adding 25 new varlink enum types.
Weak compatibility breakage (per
https://github.com/systemd/systemd/pull/40972#issuecomment-4222294318):
Varlink enum identifiers cannot contain - or +, so affected values are
underscorified on the wire. For example, "tty-force" becomes tty_force,
"kmsg+console" becomes kmsg_console.
The full list of affected values:
```
- ExecInputType: tty-force, tty-fail
- ExecOutputType: kmsg+console, journal+console
- ProtectHome: read-only
- CGroupController: bpf-firewall, bpf-devices, bpf-foreign, bpf-socket-bind, bpf-restrict-network-interfaces, bpf-bind-network-interface
- CollectMode: inactive-or-failed
- EmergencyAction: exit-force, reboot-force, reboot-immediate, poweroff-force, poweroff-immediate, soft-reboot, soft-reboot-force, kexec-force, halt-force, halt-immediate
- JobMode: replace-irreversibly, ignore-dependencies, ignore-requirements, restart-dependencies
```
We already have checks in place during compression that limit the data
we compress, so they shouldn't decompress to anything larger than
DATA_SIZE_MAX unless they've been tampered with. Let's make this
explicit and limit all our decompress_blob() calls in journal-handling
code to that limit.
One possible scenario this should prevent is when one tries to open and
verify a journal file that contains a compression bomb in its payload:
$ ls -lh test.journal
-rw-rw-r--+ 1 fsumsal fsumsal 1.2M Apr 12 15:07 test.journal
$ systemd-run --user --wait --pipe -- build-local/journalctl --verify --file=$PWD/test.journal
Running as unit: run-p682422-i4875779.service
000110: Invalid hash (00000000 vs. 11e4948d73bdafdd)
000110: Invalid object contents: Bad message
File corruption detected at /home/fsumsal/repos/@systemd/systemd/test.journal:272 (of 1249896 bytes, 0%).
FAIL: /home/fsumsal/repos/@systemd/systemd/test.journal (Bad message)
Finished with result: exit-code
Main processes terminated with: code=exited, status=1/FAILURE
Service runtime: 48.051s
CPU time consumed: 47.941s
Memory peak: 8G (swap: 0B)
Same could be, in theory, possible with just `journalctl --file=`, but
the reproducer would be a bit more complicated (haven't tried it, yet).
Lastly, the change in journal-remote is mostly hardening, as the maximum
input size to decompress_blob() there is mandated by MHD's connection
memory limit (set to JOURNAL_SERVER_MEMORY_MAX which is 128 KiB at the
time of writing), so the possible output size there is already quite
limited (e.g. ~800 - 900 MiB for xz-compressed data).
We have a test failure where the testsuite is calling
sd_varlink_idl_parse() with *ret being NULL. This is now an
assert error. So we could either fix the test or fix the code
Given that it seems genuinely useful to run sd_varlink_idl_parse()
without *ret to e.g. just check if the idl is valid I opted to
fix the code.
The updated parsing_hacks.h file uncovered a bunch of extra
things that the check-pointer-deref coccinelle script flags.
This commit fixes them to make the tree check-pointer-deref clean.
1024 connections per UID is unnecessarily generous, so let's scale this
down a bit. D-Bus defaults to 256 connections per UID, but let's be even
more conservative and go with 128.
When JSON_STREAM_INPUT_SENSITIVE is set before the first read,
input_buffer is NULL, input_buffer_size is 0, and input_buffer_index
is 0. The old condition '!INPUT_SENSITIVE && index == 0' would route
this case into the else branch which calls memcpy() with a NULL source
pointer, which is undefined behavior even when the length is zero, and
is caught by UBSan.
Fix by checking input_buffer_index == 0 first, then allowing the
GREEDY_REALLOC fast path also when input_buffer_size == 0, since
there is no sensitive data to protect from realloc() copying in that
case. The else branch is now only entered when there is actual data
to copy (input_buffer_size > 0), guaranteeing input_buffer is
non-NULL.
Follow-up for 6b1a59d594
Introduces JsonStream, a generic transport layer for JSON-line message
exchange over a pair of file descriptors. It owns the input/output
buffers, SCM_RIGHTS fd passing, the deferred output queue, the
read/write/parse step functions, sd-event integration (input/output/time
event sources), the idle timeout machinery, and peer credential caching,
but knows nothing about the specific JSON protocol on top — the consumer
drives its state machine via phase/dispatch callbacks supplied at
construction.
sd-varlink is reworked to delegate the entire transport layer to a
JsonStream owned by sd_varlink. The varlink struct drops every
transport-related field (input/output buffers and fds, output queue,
fd-passing state, ucred/pidfd cache, prefer_read/write fallback, idle
timeout, description, event sources) — all of that lives in JsonStream
now. What remains in sd_varlink is the varlink-protocol state machine
(state, n_pending, current/previous/sentinel, server linkage, peer
credentials accounting, exec_pidref, the varlink-specific quit and defer
sources) and a thin wrapper layer over the JsonStream API. The
should_disconnect / get_timeout / get_events / wait helpers all live in
JsonStream now and are driven by a JsonStreamPhase the consumer reports
via its phase callback.
Define proper varlink enum types for unit configuration settings that
are part of the user-facing API (values users/clients can select).
This replaces SD_VARLINK_STRING with SD_VARLINK_DEFINE_FIELD_BY_TYPE
for these fields, giving them strong type semantics in the IDL.
Enum types added for ExecContext (ExecInputType, ExecOutputType,
ExecUtmpMode, ExecPreserveMode, ExecKeyringMode, MemoryTHP,
ProtectProc, ProcSubset, ProtectSystem, ProtectHome, PrivateTmp,
PrivateUsers, ProtectHostname, ProtectControlGroups, PrivatePIDs,
PrivateBPF), CGroupContext (CGroupDevicePolicy, ManagedOOMMode,
ManagedOOMPreference, CGroupPressureWatch, NFTSetSource, NFProto,
BPFCGroupAttachType, CGroupController), and UnitContext (CollectMode,
EmergencyAction, JobMode).
Engine-reported runtime state fields (Type, LoadState, ActiveState,
FreezerState, SubState, UnitFileState) remain as strings since only
the engine selects those values.
As mentioned in the grandfather commit, I want to use the _many
suffix for freeing of the contents of an array, so the functions
to free the array to get the suffix _array.
newa(t, n) already allocates sizeof(t) * n bytes, so previously we'd
actually allocate sizeof(t) * sizeof(t) * n bytes, which is ~16x more
(on x86_64) that we actually needed.
This is probably an oversight from a tree-wide change in
6e9417f5b4 that replaced alloca() with
newa().
Follow-up for 6e9417f5b4.
With the old version there was a potential connection count leak if
either of the two hashmap operations in count_connection() failed. In
that case we'd return from sd_varlink_server_add_connection_pair()
_before_ attached the sd_varlink_server object to an sd_varlink object,
and since varlink_detach_server() is the only place where the connection
counter is decremented (called through sd_varlink_close() in various
error paths later _if_ the "server" object is not null, i.e. attached to
the sd_varlink object) we'd "leak" a connection every time this
happened. However, the potential of abusing this is very theoretical,
as one would need to hit OOM every time either of the hashmap operations
was executed for a while before exhausting the connection limit.
Let's just increment the connection counter after any potential error
path, so we don't have to deal with potential rollbacks.
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
When there is a potential protocol upgrade we need to be careful that
we do not read beyond our json message as the custom protocol may be
anything. This was archived via a byte-by-byte read. This is of course
very inefficient. So this commit moves to use MSG_PEEK to find the
boundary of the json message instead. This makes the performance hit
a lot smaller.
Thanks to Lennart for suggesting this.
When the server side of a varlink connection supports connection
upgrades we need to go into single byte-read mode to avoid the
risk of a client that sends the json to protocol upgrade and then
immediately the custom protocol payload. This commit implements
this.
The next step is using MSG_PEEK to avoid the single-byte overhead.
This commit tweaks the API of sd_varlink_call_and_upgrade and
sd_varlink_reply_and_upgrade to return two independent fds even
if the internal {input,output}_fd are the same (e.g. a socket).
This makes the external API easier as there is no longer the risk
of double close. The sd_varlink_call_and_upgrade() is not in a
released version of systemd yet so I presume it is okay to update
it still.
This also allowed some simplifications in varlinkctl.c now that
the handling is easier.
Extract the fd-handling logic from sd_varlink_call_and_upgrade() into a
shared static helper so that it can be reused by the upcoming server-side
sd_varlink_reply_and_upgrade().
`print_list()` and `print_info()` used `RETRIEVE()` to `strndup()` the
entire
`COREDUMP` field into a heap-allocated string, only to check whether it
exists.
With `sd_journal_set_data_threshold(j, 0)` in `print_info()`, this
copies the
full coredump binary (potentially hundreds of MB) to heap just to print
"Storage: journal".
This PR:
1. Makes `sd_journal_get_data()` output parameters optional
(`NULL`-safe), so
callers can do pure existence checks without receiving the data.
2. Short-circuits `maybe_decompress_payload()` after
`decompress_startswith()`
succeeds when neither output pointer is requested, skipping full blob
decompression for compressed journal entries.
3. Switches coredumpctl to pass `NULL, NULL` for the existence checks
instead
of heap-copying via `RETRIEVE()`.
Daan De Meyer
Christian Brauner
Lennart Poettering
Zbigniew Jędrzejewski-Szmek
Frantisek Sumsal
Yu Watanabe
Christian Brauner
Michael Vogt
Luca Boccassi
Ivan Kruglov