Introduce support for running code in fibers (#39771)

Traditionally, asynchronous programming in systemd has been achieved
using
sd-event along with the asynchronous interfaces of sd-bus and
sd-varlink.
This works well when the system is reacting to events and all code
triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback
can
use the stack as usual, declaring local state and ensuring proper
cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.

However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block
execution
within the callback, we can't directly invoke a long-running operation
and
wait for its result without introducing complexities. Instead, we need
to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.

This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along
as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope
may
be prematurely cleaned up. Therefore, the struct must store copies of
the
local variables or ensure proper reference counting to prevent premature
resource cleanup.

When multiple long-running operations need to be initiated within a
loop,
the complexity grows further. We must introduce additional shared state
to
track the completion of all operations before we can run any code that
depends on their results.

Furthermore, since the daemon may be shut down at any time, we must
track
the lifecycle of each long-running operation in the global Manager
struct,
ensuring proper cleanup even when stack unwinding can no longer manage
the
resources for us.

Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s
running
synchronously, but with the ability to yield control at predefined
points,
such as when waiting for long-running tasks to complete.

With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while
waiting
for the long-running operation to finish and then "resume" the operation
once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.

This commit introduces the ability to execute long-running operations in
a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve
the
handling of complex workflows, making the code easier to write and
maintain.

The implementation is based on ucontext.h's makecontext() (with a
fallback
to the venerable sigaltstack() approach on musl),
sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can
switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext()
because the
latter forcibly saves/restores a per context signal mask every time it
is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and
maintain
a per thread signal mask, which makes much more sense than having a per
fiber
signal mask.

The default stack size is the same as a regular thread. Because we 
use mmap() to allocate the stack, the memory won't actually be used
until it
is paged in by the kernel, so we don't actually use 8MB per fiber.

To integrate fibers with the event loop, each fiber is assigned a
deferred
event source which resumes the fiber when enabled. The deferred event
source
is oneshot by default so the fiber will run immediately until it yields
or
suspends. If it yields, the deferred event source is enabled again
(oneshot)
immediately. If it suspends, before it suspends, one or more event
sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for
completes.

Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or
any
error value from the async operation that caused the fiber to resume.

This is also how fiber cancellation is implemented. When a fiber is
cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the
fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup
to
happen automatically) and finish.

Instead of having applications work directly with fibers, we hide them
behind
a generic futures interface to represent long-running operations,
regardless of
whether those operations are running on a fiber or not. Aside from
fibers, the
futures library (sd-future) will for example allow waiting for sd-event
sources
and doing sd-bus calls in the background as well. Fibers can suspend
until a
future is ready with sd_fiber_await() or by having the future wake up
the fiber
explicitly in its callback. A future always defaults to waking up the
current
fiber.

Each future kind plugs into the library by providing an sd_future_ops
vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it
via
sd_future_get_private().

A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to
that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from
a
fiber via sd_fiber_await() (synchronous-looking fiber code).
sd_fiber_await()
is itself built on a "wait future" that resolves when its target
resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber
callers
can chain futures without involving a fiber.

Cancellation is cooperative: sd_future_cancel() invokes the future
impl's
cancel callback, which is responsible for tearing down its work and
ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from
sd_fiber_yield()/sd_fiber_suspend()
mentioned above.

Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new()
—
take a self-reference on their future so they outlive the caller's
scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds
would
risk silent leaks for kinds that may never resolve.

Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal
sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an
unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.

The code lives in libsystemd as sd-future (not exported) for the
following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of
it

The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.

src/basic/architecture.h

@@ -242,4 +242,10 @@ Architecture uname_architecture(void);
 #  error "Please register your architecture here!"
 #endif
 
+#if defined(__hppa__) || defined(__hppa64__)
+#  define STACK_GROWS_UP 1
+#else
+#  define STACK_GROWS_UP 0
+#endif
+
 DECLARE_STRING_TABLE_LOOKUP(architecture, Architecture);

src/basic/basic-forward.h

@@ -112,10 +112,13 @@ typedef enum UnitNameMangle UnitNameMangle;
 typedef enum UnitType UnitType;
 typedef enum WaitFlags WaitFlags;
 
+typedef struct Fiber Fiber;
+typedef struct FiberOps FiberOps;
 typedef struct Hashmap Hashmap;
 typedef struct HashmapBase HashmapBase;
 typedef struct IteratedCache IteratedCache;
 typedef struct Iterator Iterator;
+typedef struct LogContext LogContext;
 typedef struct OrderedHashmap OrderedHashmap;
 typedef struct OrderedSet OrderedSet;
 typedef struct Set Set;

src/basic/fiber-ops.c

@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+#include <poll.h>
+#include <threads.h>
+#include <unistd.h>
+
+#include "errno-util.h"
+#include "fiber-ops.h"
+
+static thread_local const FiberOps *fiber_ops = NULL;
+
+bool fiber_ops_is_set(void) {
+        return fiber_ops != NULL;
+}
+
+void fiber_ops_set(const FiberOps *ops) {
+        fiber_ops = ops;
+}
+
+int fiber_ops_ppoll(struct pollfd *fds, size_t n_fds, const struct timespec *timeout, const sigset_t *sigmask) {
+        if (fiber_ops)
+                return fiber_ops->ppoll(fds, n_fds, timeout, sigmask);
+
+        return RET_NERRNO(ppoll(fds, n_fds, timeout, sigmask));
+}
+
+ssize_t fiber_ops_read(int fd, void *buf, size_t count) {
+        if (fiber_ops)
+                return fiber_ops->read(fd, buf, count);
+
+        ssize_t n = read(fd, buf, count);
+        return n < 0 ? -errno : n;
+}
+
+ssize_t fiber_ops_write(int fd, const void *buf, size_t count) {
+        if (fiber_ops)
+                return fiber_ops->write(fd, buf, count);
+
+        return RET_NERRNO(write(fd, buf, count));
+}
+
+sd_future* fiber_ops_timeout(uint64_t timeout) {
+        assert(fiber_ops);
+
+        return fiber_ops->timeout(timeout);
+}
+
+sd_future* fiber_ops_cancel_wait_unref(sd_future *f) {
+        assert(fiber_ops);
+
+        return fiber_ops->cancel_wait_unref(f);
+}

src/basic/fiber-ops.h

@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: LGPL-2.1-or-later */
+#pragma once
+
+#include "basic-forward.h"
+
+typedef struct sd_future sd_future;
+
+/* Hooks installed on a fiber so that functions in src/basic can transparently defer to the suspending
+ * variants in sd-future when invoked from a running fiber. Populated by sd_fiber_new() with pointers to the
+ * implementations in fiber-ops.c. */
+typedef struct FiberOps {
+        int (*ppoll)(struct pollfd *fds, size_t n_fds, const struct timespec *timeout, const sigset_t *sigmask);
+        ssize_t (*read)(int fd, void *buf, size_t count);
+        ssize_t (*write)(int fd, const void *buf, size_t count);
+        sd_future* (*timeout)(uint64_t timeout);
+        sd_future* (*cancel_wait_unref)(sd_future *f);
+} FiberOps;
+
+bool fiber_ops_is_set(void);
+void fiber_ops_set(const FiberOps *fiber_ops);
+
+int fiber_ops_ppoll(struct pollfd *fds, size_t n_fds, const struct timespec *timeout, const sigset_t *sigmask);
+ssize_t fiber_ops_read(int fd, void *buf, size_t count);
+ssize_t fiber_ops_write(int fd, const void *buf, size_t count);
+
+/* Mirror of SD_FIBER_TIMEOUT() for code under src/basic that doesn't include sd-future.h: dispatches
+ * through FiberOps so the actual sd_fiber_timeout() implementation lives in libsystemd. */
+sd_future* fiber_ops_timeout(uint64_t timeout);
+sd_future* fiber_ops_cancel_wait_unref(sd_future *f);
+DEFINE_TRIVIAL_CLEANUP_FUNC(sd_future*, fiber_ops_cancel_wait_unref);
+
+#define FIBER_OPS_TIMEOUT(timeout) _FIBER_OPS_TIMEOUT(UNIQ, (timeout))
+#define _FIBER_OPS_TIMEOUT(uniq, timeout)                                                                                               \
+        _unused_ _cleanup_(fiber_ops_cancel_wait_unrefp) sd_future *UNIQ_T(_fot_, uniq) = fiber_ops_timeout(timeout)

src/basic/io-util.c

@@ -1,15 +1,26 @@
 /* SPDX-License-Identifier: LGPL-2.1-or-later */
 
+#include <fcntl.h>
 #include <poll.h>
 #include <stdio.h>
 #include <string.h>
+#include <sys/epoll.h>          /* IWYU pragma: keep */
 #include <time.h>
 #include <unistd.h>
 
 #include "errno-util.h"
+#include "fiber-ops.h"
 #include "io-util.h"
 #include "time-util.h"
 
+/* EPOLL_POLL_COMMON_MASK in io-util.h treats POLL* and EPOLL* as interchangeable; verify it. */
+assert_cc((uint32_t) POLLIN    == EPOLLIN);
+assert_cc((uint32_t) POLLOUT   == EPOLLOUT);
+assert_cc((uint32_t) POLLERR   == EPOLLERR);
+assert_cc((uint32_t) POLLHUP   == EPOLLHUP);
+assert_cc((uint32_t) POLLPRI   == EPOLLPRI);
+assert_cc((uint32_t) POLLRDHUP == EPOLLRDHUP);
+
 int flush_fd(int fd) {
         int count = 0;
 
@@ -60,25 +71,44 @@ ssize_t loop_read(int fd, void *buf, size_t nbytes, bool do_poll) {
         if (nbytes > (size_t) SSIZE_MAX)
                 return -EINVAL;
 
+        /* do_poll == false means "don't wait, return what we have if EAGAIN". If the fd is already
+         * non-blocking, read() can't block the thread, so the non-fiber path satisfies that semantic
+         * correctly even from a fiber. Only use the fiber path when the fd is blocking (where read()
+         * would otherwise block the entire event loop). */
+        int flags = 0;
+        if (fiber_ops_is_set() && !do_poll) {
+                flags = fcntl(fd, F_GETFL);
+                if (flags < 0)
+                        return -errno;
+        }
+
         do {
                 ssize_t k;
 
-                k = read(fd, p, nbytes);
-                if (k < 0) {
-                        if (errno == EINTR)
-                                continue;
+                if (fiber_ops_is_set() && (do_poll || !FLAGS_SET(flags, O_NONBLOCK))) {
+                        /* On a fiber the read op suspends on EAGAIN until data is available, so we don't
+                         * need a separate poll step or the do_poll knob. */
+                        k = fiber_ops_read(fd, p, nbytes);
+                        if (k < 0)
+                                return n > 0 ? n : k;
+                } else {
+                        k = read(fd, p, nbytes);
+                        if (k < 0) {
+                                if (errno == EINTR)
+                                        continue;
 
-                        if (errno == EAGAIN && do_poll) {
+                                if (errno == EAGAIN && do_poll) {
 
-                                /* We knowingly ignore any return value here,
-                                 * and expect that any error/EOF is reported
-                                 * via read() */
+                                        /* We knowingly ignore any return value here,
+                                         * and expect that any error/EOF is reported
+                                         * via read() */
 
-                                (void) fd_wait_for_event(fd, POLLIN, USEC_INFINITY);
-                                continue;
+                                        (void) fd_wait_for_event(fd, POLLIN, USEC_INFINITY);
+                                        continue;
+                                }
+
+                                return n > 0 ? n : -errno;
                         }
-
-                        return n > 0 ? n : -errno;
                 }
 
                 if (k == 0)
@@ -128,6 +158,37 @@ int loop_write_full(int fd, const void *buf, size_t nbytes, usec_t timeout) {
                 p = buf;
         }
 
+        /* timeout == 0 means "don't wait, return -EAGAIN if not ready". If the fd is already
+         * non-blocking, write() can't block the thread, so the non-fiber path satisfies that
+         * semantic correctly even from a fiber. Only use the fiber path when the fd is blocking
+         * (where write() would otherwise block the entire event loop). */
+        int flags = 0;
+        if (fiber_ops_is_set() && timeout == 0) {
+                flags = fcntl(fd, F_GETFL);
+                if (flags < 0)
+                        return -errno;
+        }
+
+        if (fiber_ops_is_set() && !FLAGS_SET(flags, O_NONBLOCK)) {
+                /* On a fiber the write op suspends on EAGAIN until the fd is writable; honor the
+                 * caller's timeout via a deadline scope. */
+                FIBER_OPS_TIMEOUT(timestamp_is_set(timeout) ? timeout : USEC_INFINITY);
+
+                while (nbytes > 0) {
+                        ssize_t k = fiber_ops_write(fd, p, nbytes);
+                        if (k < 0)
+                                return (int) k;
+                        if (_unlikely_(nbytes > 0 && k == 0)) /* Can't really happen */
+                                return -EIO;
+
+                        assert((size_t) k <= nbytes);
+                        p += k;
+                        nbytes -= k;
+                }
+
+                return 0;
+        }
+
         /* When timeout is 0 or USEC_INFINITY this is not used. But we initialize it to a sensible value. */
         end = timestamp_is_set(timeout) ? usec_add(now(CLOCK_MONOTONIC), timeout) : USEC_INFINITY;
 
@@ -211,11 +272,9 @@ int ppoll_usec_full(struct pollfd *fds, size_t n_fds, usec_t timeout, const sigs
         if (n_fds == 0 && timeout == 0)
                 return 0;
 
-        r = ppoll(fds, n_fds, timeout == USEC_INFINITY ? NULL : TIMESPEC_STORE(timeout), ss);
-        if (r < 0)
-                return -errno;
-        if (r == 0)
-                return 0;
+        r = fiber_ops_ppoll(fds, n_fds, timeout == USEC_INFINITY ? NULL : TIMESPEC_STORE(timeout), ss);
+        if (r <= 0)
+                return r;
 
         for (size_t i = 0, n = r; i < n_fds && n > 0; i++) {
                 if (fds[i].revents == 0)

src/basic/io-util.h

@@ -3,6 +3,12 @@
 
 #include "basic-forward.h"
 
+/* The intersection of poll() and epoll_wait() event masks. Linux defines POLL* and EPOLL* with the
+ * same numeric values for these — see the assert_cc()s in io-util.c — so this mask can be used
+ * interchangeably as a `revents` (poll) or `events` (epoll) bitset. */
+#define EPOLL_POLL_COMMON_MASK \
+        (EPOLLIN | EPOLLOUT | EPOLLERR | EPOLLHUP | EPOLLPRI | EPOLLRDHUP)
+
 int flush_fd(int fd);
 
 ssize_t loop_read(int fd, void *buf, size_t nbytes, bool do_poll);

src/basic/log-context.c

@@ -177,6 +177,14 @@ size_t log_context_num_fields(void) {
         return _log_context_num_fields;
 }
 
+void log_context_swap(LogContext **log_context, size_t *num_fields) {
+        assert(log_context);
+        assert(num_fields);
+
+        SWAP_TWO(_log_context, *log_context);
+        SWAP_TWO(_log_context_num_fields, *num_fields);
+}
+
 void _reset_log_level(int *saved_log_level) {
         assert(saved_log_level);
 

src/basic/log-context.h

@@ -66,6 +66,8 @@ size_t log_context_num_contexts(void);
 /* Returns the number of fields in all attached log contexts. */
 size_t log_context_num_fields(void);
 
+void log_context_swap(LogContext **log_context, size_t *num_fields);
+
 void _reset_log_level(int *saved_log_level);
 
 #define _LOG_CONTEXT_SET_LOG_LEVEL(level, l) \

src/basic/log.c

@@ -87,6 +87,12 @@ bool _log_message_dummy = false; /* Always false */
                 }                                                       \
         } while (false)
 
+void log_prefix_swap(const char **prefix) {
+        assert(prefix);
+
+        SWAP_TWO(log_prefix, *prefix);
+}
+
 static void log_close_console(void) {
         /* See comment in log_close_journal() */
         (void) safe_close_above_stdio(TAKE_FD(console_fd));

src/basic/log.h

@@ -380,6 +380,8 @@ int log_syntax_parse_error_internal(
 void log_setup(void);
 
 const char* _log_set_prefix(const char *prefix, bool force);
+
+void log_prefix_swap(const char **prefix);
 static inline const char* _log_unset_prefixp(const char **p) {
         assert(p);
         _log_set_prefix(*p, true);

src/basic/meson.build

@@ -36,6 +36,7 @@ basic_sources = files(
         'ether-addr-util.c',
         'extract-word.c',
         'fd-util.c',
+        'fiber-ops.c',
         'fileio.c',
         'filesystems.c',
         'format-ifname.c',

src/basic/pidref.c

@@ -7,6 +7,7 @@
 #include "alloc-util.h"
 #include "errno-util.h"
 #include "fd-util.h"
+#include "fiber-ops.h"
 #include "format-util.h"
 #include "hash-funcs.h"
 #include "io-util.h"
@@ -466,16 +467,28 @@ int pidref_wait_for_terminate_full(PidRef *pidref, usec_t timeout, siginfo_t *re
         if (pidref->pid == 1 || pidref_is_self(pidref))
                 return -ECHILD;
 
-        if (timeout != USEC_INFINITY && pidref->fd < 0)
+        if (pidref->fd < 0 && (timeout != USEC_INFINITY || fiber_ops_is_set()))
                 return -ENOMEDIUM;
 
         usec_t ts = timeout == USEC_INFINITY ? USEC_INFINITY : usec_add(now(CLOCK_MONOTONIC), timeout);
 
+        /* Poll the pidfd before waitid() if either there's a finite timeout (so we can honor it) or
+         * we're on a fiber (so fd_wait_for_event() can suspend us instead of blocking the event loop
+         * inside waitid()). Otherwise let waitid() block directly. The precondition above guarantees
+         * pidref->fd >= 0 in both cases. */
+        bool poll_first = ts != USEC_INFINITY || fiber_ops_is_set();
+
         for (;;) {
-                if (ts != USEC_INFINITY) {
-                        usec_t left = usec_sub_unsigned(ts, now(CLOCK_MONOTONIC));
-                        if (left == 0)
-                                return -ETIMEDOUT;
+                if (poll_first) {
+                        usec_t left;
+
+                        if (ts == USEC_INFINITY)
+                                left = USEC_INFINITY;
+                        else {
+                                left = usec_sub_unsigned(ts, now(CLOCK_MONOTONIC));
+                                if (left == 0)
+                                        return -ETIMEDOUT;
+                        }
 
                         r = fd_wait_for_event(pidref->fd, POLLIN, left);
                         if (r == 0)