This introduce a vtable for journal tagging feature in sd-journal,
and makes libgcrypt dependent features loaded by users (journald,
journalctl, journal-remote, and unit tests) when necessary.
Then, move several components for journal tagging in JournalFile
to JournalAuthContext.
This also introduces wrapper functions that checks gcrypt support.
Fixes a family of OOB reads in the trie walker where attacker-controlled
offsets are loaded from a (possibly hostile) hwdb.bin and added to
hwdb->map without first checking that the resulting pointer stays inside
the mapping. Crash sites:
- trie_fnmatch_f sd-hwdb.c:187 (#42340)
- trie_search_f sd-hwdb.c:233 (#42341)
- trie_children_cmp_f /bsearch sd-hwdb.c:94 (#42342)
- hwdb_add_property sd-hwdb.c:121 (#42343)
- trie_node_from_off (UBSAN) sd-hwdb.c:83
- trie_fnmatch_f stack overflow on cyclic children (CIFuzz finding)
Commit b45a897edc ("hwdb: reject out-of-bounds fnmatch prefixes")
plugged the first site only at the prefix_off-content level, but the
crash on the read of node->prefix_off itself still happens when node is
already OOB. This is the structural fix: a single hwdb_at() helper
validates that [off, off + size) lies inside the mapping, and
trie_node_from_off() / trie_string() return NULL on OOB; every caller
treats the result as nullable and surfaces -EBADMSG.
trie_fnmatch_f() additionally caps its recursion depth at 2048; without
the cap a corrupt trie whose child offsets form a cycle (or just a deep
linear chain) drives the parser into stack-overflow rather than
returning -EBADMSG (caught by CIFuzz address sanitizer on this branch).
Additions:
- A fuzz-hwdb libFuzzer harness that drives sd_hwdb_new_from_path,
sd_hwdb_get, sd_hwdb_seek, and sd_hwdb_enumerate on attacker bytes.
- Five hand-crafted unit tests in test-sd-hwdb.c (one per crash
bucket, plus the cyclic-trie recursion case) that build a malformed
in-memory hwdb.bin and assert -EBADMSG rather than SIGSEGV /
stack-overflow.
- Regression corpus files under test/fuzz/fuzz-hwdb/ pinning each
fixed bucket, including the CIFuzz stack-overflow reproducer.
Closes#42340.
Closes#42341.
Closes#42342.
Closes#42343.
Reported-by: AI-assisted libFuzzer campaign
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Two changes to teach sd-bus how to behave when called from a fiber, in
order of increasing depth:
2. sd_bus_call() now redirects to a new bus_call_suspend() helper when
the caller is a fiber whose event loop is the same one the bus is
attached to. The plain bus_poll() path serializes all bus traffic on
the slot's reply (only one method call can be in flight per
sd_bus*), which would defeat the point of running multiple fibers
against one bus. bus_call_suspend() builds on the async sd-bus API:
it wraps the call in a new BusFuture (sd-bus/bus-future.{c,h}) that
resolves when the reply or method-error arrives, lets the fiber
await that future, and surfaces the reply to the caller via
future_get_bus_reply(). Because the futures live on the event loop
rather than a per-bus slot, multiple fibers can drive concurrent
method calls against the same bus.
3. A new private SD_BUS_VTABLE_METHOD_FIBER flag dispatches a vtable
method handler on its own fiber, so handlers are free to use
sd_bus_call() against the same bus, sd_fiber_sleep(), loop_read(),
etc. without stalling the event loop for other connections or
handlers. The flag stays out of sd-bus-vtable.h (its bit value is
reserved there to prevent collisions) — the fiber runtime is a
systemd-internal implementation detail.
Lifecycle of fiber-dispatched handlers is tracked on the bus itself: a
new bus->fiber_futures set holds a ref to each in-flight handler.
bus_enter_closing() cancels every entry and process_closing() returns
with the bus still in CLOSING state until the set drains, so we can be
sure no fiber handler outlives the bus. bus_fiber_resolved() removes
the entry on completion. bus_free()'s assert(set_isempty()) makes the
invariant load-bearing.
Note that plain sd_bus_call() already works correctly on a fiber as it
calls ppoll_usec() which has already been modified to suspend when
running on a fiber.
To exercise these changes the existing thread-based client/server
sd-bus tests (test-bus-chat, test-bus-objects, test-bus-peersockaddr,
test-bus-server, test-bus-watch-bind) are migrated to fibers, and a
new test-bus-fiber is added that covers SD_BUS_VTABLE_METHOD_FIBER —
including handlers that issue nested sd_bus_call() on the same bus, the
cancel-on-close path, and concurrent dispatches across multiple fibers.
sd_event_run() blocks the calling thread on the event loop's epoll fd
until something happens. When the caller is a fiber, that's the wrong
behaviour: blocking the thread also stalls every other fiber and the
outer event loop driving them. The most common way to hit this is a
fiber that creates its own inner event loop (e.g. a server-style fiber
that wants to dispatch its own sources independently of whatever loop
the test or supervising fiber is running on) — with the existing
implementation the inner sd_event_run() would hold the thread while the
outer scheduler should be free to advance other fibers.
Add an event_run_suspend() variant in sd-event/event-future.c that
performs the same prepare/wait/dispatch dance, but when the fast path
finds nothing ready it (a) creates an IO future watching the inner
event loop's epoll fd on the *outer* event loop, (b) optionally creates
a time future for the timeout, and (c) suspends the fiber. When either
future fires the fiber is resumed and the prepare/wait/dispatch sequence
runs once more to actually dispatch what's pending. sd_event_run()
checks sd_fiber_is_running() and delegates to this variant when on a
fiber; profile_delays accounting is intentionally skipped on that path
since the underlying prepare/wait/dispatch primitives already account
for themselves.
Some helpers in src/basic — ppoll_usec_full() (used by fd_wait_for_event()),
loop_read(), loop_read_exact(), loop_write_full() and
pidref_wait_for_terminate_full() — block the calling thread. That's the
right behaviour outside a fiber but not inside one, where blocking the
thread also stalls every other fiber running on the same event loop.
Rewriting every caller to pick a fiber or non-fiber variant explicitly
would be a lot of churn and would split otherwise-shared code paths in
two.
Instead, the helpers detect at runtime whether they're running on a fiber
and dispatch to a suspending variant when they are. FiberOps in
fiber-ops.h holds five function pointers (ppoll, read, write, timeout,
cancel_wait_unref); a fiber_ops global constant is populated whenever we
enter a fiber with functions that delegate to suspending variants of common
syscalls. With this approach, the variants themselves stay in libsystemd
which is required because they make use of sd-event.
- loop_read()/loop_read_exact() take the fiber read hook on a fiber
unless the caller asked for a non-blocking attempt (do_poll=false) and
the fd is already non-blocking — in that case we fall through to read()
to preserve the existing return-EAGAIN-immediately semantic. The hook
itself suspends on EAGAIN until data is available, so neither the
do_poll knob nor the explicit fd_wait_for_event() retry loop are
needed on the fiber path.
- loop_write_full() likewise takes the fiber write hook on a fiber,
except when timeout=0 with an already-non-blocking fd (preserving the
fast-return-EAGAIN semantic). The fiber path runs inside a
FIBER_OPS_TIMEOUT() scope so the caller's timeout is honoured via a
deadline future, mirroring SD_FIBER_TIMEOUT() but reachable from
src/basic without pulling in sd-future.h.
- pidref_wait_for_terminate_full() polls the pidfd via fd_wait_for_event()
before each waitid() when either a finite timeout is set or we're on a
fiber, and requires pidref->fd >= 0 in those cases (returning
-ENOMEDIUM otherwise — extending the rule that already applied to
finite timeouts). The poll suspends the fiber via the ppoll hook above;
the subsequent waitid() doesn't block because the pidfd is already
signalled.
Add a family of sd_fiber_*() I/O wrappers that, when called from a
fiber, behave like blocking I/O from the caller's perspective but
yield to the event loop instead of blocking the thread:
sd_fiber_read / sd_fiber_write
sd_fiber_readv / sd_fiber_writev
sd_fiber_recv / sd_fiber_send
sd_fiber_connect
sd_fiber_recvmsg / sd_fiber_sendmsg
sd_fiber_recvfrom / sd_fiber_sendto
sd_fiber_accept
sd_fiber_ppoll
Most of them share a single helper, fiber_io_operation(), which when
invoked outside a fiber falls through to the underlying syscall
directly, preserving the regular blocking behaviour. Inside a fiber
the helper flips the fd to non-blocking (restoring its original mode
on return), tries the syscall once on the fast path, and on EAGAIN/
EWOULDBLOCK creates an sd-event-backed IO future via future_new_io(),
suspends the fiber, and retries the syscall once the event source
fires.
future_new_io() itself is added to sd-event/event-future.{c,h} as a
new IoFuture kind. It wraps sd_event_add_io() into an sd_future:
oneshot enable, EPOLLERR translated via SO_ERROR (suppressed for
non-sockets), and the fd duplicated with F_DUPFD_CLOEXEC to avoid
EEXIST when multiple sources watch the same descriptor.
Together these let fiber-using code write straight-line socket and
pipe I/O without bundling state into callbacks.
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
- test-random-util is reworked to not use sqrt()
- pretty-print.c inlines ceil() so libm doesn't have
to be linked into libshared
- We add fno-math-errno to allow inlining of more math
functions by not requiring standard math functions to
set errno on invalid input.
Our baseline glibc is 2.34, which merged libdl, libpthread (the
dependency('threads') target), and librt into libc. Empty .so/.a stubs
remain for backward compatibility with old binaries, but new builds
resolve dl_*, pthread_*, mq_*, timer_*, etc. directly from libc.
On musl the same libraries are likewise empty stubs.
Drop the libdl, threads, and librt entries from every meson.build, and
remove the now-stale 'Libs.private: -lrt -pthread' from libudev.pc.in
since both flags resolve to empty link-time stubs on glibc 2.34+ and
musl.
Verified with readelf -d that libsystemd.so, libudev.so, and systemd no
longer carry DT_NEEDED entries for libdl/libpthread/librt.
The test only uses 9 symbols (5 from glib, 4 from libdbus) for its
interop checks; dlopen them at runtime so the binary no longer carries a
hard link-time dependency on either library. Headers are still pulled in
through the *_cflags partial dependencies for the type declarations.
While we're at it, drop the compat glue for glib 2.36 which is long obsolete
at this point.
Merge the two blocks adding tests, since there seems to be
no obvious reason to have two separate blocks, as they both
contain tests from the same libraries.
Introduces JsonStream, a generic transport layer for JSON-line message
exchange over a pair of file descriptors. It owns the input/output
buffers, SCM_RIGHTS fd passing, the deferred output queue, the
read/write/parse step functions, sd-event integration (input/output/time
event sources), the idle timeout machinery, and peer credential caching,
but knows nothing about the specific JSON protocol on top — the consumer
drives its state machine via phase/dispatch callbacks supplied at
construction.
sd-varlink is reworked to delegate the entire transport layer to a
JsonStream owned by sd_varlink. The varlink struct drops every
transport-related field (input/output buffers and fds, output queue,
fd-passing state, ucred/pidfd cache, prefer_read/write fallback, idle
timeout, description, event sources) — all of that lives in JsonStream
now. What remains in sd_varlink is the varlink-protocol state machine
(state, n_pending, current/previous/sentinel, server linkage, peer
credentials accounting, exec_pidref, the varlink-specific quit and defer
sources) and a thin wrapper layer over the JsonStream API. The
should_disconnect / get_timeout / get_events / wait helpers all live in
JsonStream now and are driven by a JsonStreamPhase the consumer reports
via its phase callback.
Then, move syscall definitions to the wrapper, and prototypes are moved
to relevant headers.
This also adds checks for add_key() and request_key(), as one day
glibc may be going to add some of them separatedly.
The check for fspick in meson.build is dropped, as it is currently
unused in our code.
This also moves
- basic/missing_bpf.h -> include/override/linux/bpf.h,
- basic/missing_keyctl.h -> include/override/linux/keyctl.h.
[1] says:
> Since 0.60.0 the name argument is optional and defaults to the basename of
> the first output
We specify >= 0.62 as the supported version, so drop the duplicate name in all cases
where it is the same as outputs[0], i.e. almost all cases.
[1] https://mesonbuild.com/Reference-manual_functions.html#custom_target
Instead of using run-clang-tidy.py with its own scheduling, let's
just gather a list of source files ourselves and then use that to
add a unit test for each source file that runs clang-tidy on the
source file.
We also add a bit of logic to run clang-tidy on most header files
as well for extra coverage. This uncovered various header files that
were not standalone so this commit also includes fixes to make sure
the clang-tidy tests are all green.
We can also use this in a later commit to run clang-include-cleaner
on each source file in the same way.
meson by default adds the current source and build directory as include
directories. Because we structure our meson code by gathering a giant dict
of everything we want to do and then doing all the actual target generation
in the top level meson.build, this behavior does not make sense at all because
we end up adding the top level repository directory as an include directory
which is never what we want.
At the same time, let's also make sure the top level directory of the build
directory is not an include directory, by moving the version.h generation
into the src/version subdirectory and then adding the src/version subdirectory
of the build directory as an include directory instead of the top level
repository directory.
Making this change means that language servers such as clangd can't get
confused when they automatically insert an #include line and insert
"#include "src/basic/fs-util.h" instead of "#include "fs-util.h".
As a test, it just increases our code coverage in a fake way.
When run manually, it can be used to conveniently print what logind
thinks about various processes:
$ build/test-sd-login
sd_pid_get_session(0) → No data available
sd_pid_get_unit(0) → user@1000.service
sd_pid_get_user_unit(0) → app-ghostty-transient-5088.scope
sd_pid_get_machine_name(0) → No such file or directory
sd_pid_get_slice(0) → user-1000.slice
sd_pid_get_user_slice(0) → app.slice
sd_pid_get_owner_uid(0) → 1000
sd_pid_get_cgroup(0) → /user.slice/user-1000.slice/user@1000.service/app.slice/app-ghostty-transient-5088.scope/surfaces/556FAF50BA40.scope
$ build/test-sd-login cgroup 1
sd_pid_get_cgroup(1) → /init.scope
meson's target has a few issues:
- Runs on all source files regardless if they're included in the
build or not
- Doesn't have any dependencies on generated sources which means we
have to do a full build first before we can run clang-tidy
- Doesn't allow us to pass any extra arguments
To work around these, let's define our own clang-tidy target instead
using llvm's run-clang-tidy script. Alongside the clang-tidy target,
let's start keeping track of all generated sources which we make the
clang-tidy target depend on. We also add a new target which will only
generate source files which is useful for setting up the source tree
for running code analysis against it.
Tooling such as clang-tidy is bad at dealing with condition
compilation on the build system level instead of at the source file
level. What happens? It still tries to analyze the file and fails
horribly if the required headers aren't available. Let's work around
the issue and make things more consistent at the same time by doing
the necessary HAVE_GCRYPT checks inside of the source files instead
of doing them at the build system level.
We also add some typedefs to allow getting rid of various HAVE_GCRYPT
checks.
It's time. sd-json was already done earlier in this cycle, let's now
make sd-varlink public too.
This is mostly just a search/replace job of epical proportions.
I left some functions internal (mostly IDL handling), and I turned some
static inline calls into regular calls.
Our variables for internal libraries are named 'libfoo' for the shared lib
variant, and 'libfoo_static' for the static lib variant. The only exception was
libbasic, because we didn't have a shared variant for it. But let's rename it
for consitency. This makes the build config easier to understand.
This is preparation for making our Varlink API a public API. Since our
Varlink API is built on top of our JSON API we need to make that public
first (it's a nice API, but JSON APIs there are already enough, this is
purely about the Varlink angle).
I made most of the json.h APIs public, and just placed them in
sd-json.h. Sometimes I wasn't so sure however, since the underlying data
structures would have to be made public too. If in doubt I didn#t risk
it, and moved the relevant API to src/libsystemd/sd-json/json-util.h
instead (without any sd_* symbol prefixes).
This is mostly a giant search/replace patch.
The test-event test seems to be taking quite a bit more time than
the other 'simple tests', which usually complete in < 1s. In case
of a slower or loaded machine the default 30s timeout is not enough.
Dynamically load liblz4, libzstd and liblzma with dlopen().
This helps to reduce the size of the initrd image when these libraries
are not really needed.
Although this slightly more verbose it makes it much easier to reason
about. The code that produces the tests heavily benefits from this.
Test lists are also now sorted by test name.
A lot of tests can be defined by just their filename. Moving into their
own list keeps things simpler, especially with the next commit. It also
makes it easier to keep the lists sorted.
It had two symbols which were not actually exported because they were not
listed in libsystemd.sym. They were also entirely unused in our codebase.
I don't think it makes much sense to export just those two functions, and
it doesn't make to build a string processing library in systemd either.
History of the file shows that it was created in
faaa5728d9 'utf8: export utf8 validation functions as part of sd-bus'
and hasn't gone even one non-trivial change since then ;)
Before we had the following scheme:
mempool_enabled() would check mempool_use_allowed, and
libsystemd-shared would be linked with a .c file that provides mempool_use_allowed=true,
while other things would linked with a different .c file with mempool_use_allowed=false.
In the new scheme, mempool_enabled() itself is a weak symbol. If it's
not found, we assume false. So it only needs to be provided for libsystemd-shared,
where it can return false or true.
test-set-disable-mempool is libshared, so it gets the symbol. But then we
actually disable the mempool via envvar. mempool_enable() is called to check
its return value directly.
Yu Watanabe
jeffhuang
Daan De Meyer
Lennart Poettering
Zbigniew Jędrzejewski-Szmek
Daan De Meyer
Mike Yuan
Radoslav Kolev
Matteo Croce
Jan Janssen
Franck Bui
Luca Boccassi