Otherwise, if sd_event_add_inotify() is called with NULL for ret,
the event source is disabled and never triggered.
Fixes a bug introduced by e67d738a87.
(cherry picked from commit 0582f7cba907eba6d55b686028d145061c38d0b7)
As mentioned in a05fe6684a, glibc provides two
wrappers for epoll_pwait2() on 32-bit architectures, and sys/epoll.h,
together with features-time64.h, selects the appropriate one.
To avoid depending on symbols introduced in newer glibc releases, that
commit relied on the glibc internal macro __USE_TIME_BITS64 to determine
which wrapper to use. However, that is incorrect on some architectures.
See:
https://sourceware.org/git/?p=glibc.git;a=commit;h=a4ed0471d71739928a0d0fa3258b3ff3b158e9b9
The above glibc commit introduces a new internal macro,
__USE_TIME64_REDIRECTS, which correctly describes whether the time64
wrapper should be used. However, we cannot rely on it because it was
introduced in glibc 2.40, while our baseline remains glibc 2.34.
Moreover, even after raising the baseline, relying on glibc internal
implementation details is undesirable.
Instead, call the epoll_pwait2 syscall directly. This avoids any
dependency on glibc wrapper selection logic while preserving the original
goal of avoiding newer glibc symbols.
As a side effect, this also enables epoll_pwait2() on musl, which does not
currently provide a libc wrapper for it.
Currently, epoll_pwait2() is only used by sd-event, so this is also the
simplest solution.
When a rate limit is armed, for example via StartLimitInterval=, and the
machine is suspended, the rate limit's "clock" is suspended as well,
since the elapse checks use CLOCK_MONOTONIC. This then causes unexpected
situations where a rate limit armed via StartLimitInterval=1h, followed
by a 10 hour suspend, would elapse after 11 hours total.
Let's avoid this by switching the rate limits to CLOCK_BOOTTIME, which
works the same as CLOCK_MONOTONIC, but accounts for the time spent in
suspend as well.
There's one slight concern when it comes to upgrade path - the old
"begin" value of the rate limit is stored as CLOCK_MONOTONIC, but after
upgrading systemd and serializing/deserializing the state it will be
suddenly compared against now(CLOCK_BOOTTIME), which might cause some
rate limits to elapse "prematurely". But this is just a one-time thing,
after which the rate limit timers should re-assess themselves.
Resolves: #42912
sd_event_run() blocks the calling thread on the event loop's epoll fd
until something happens. When the caller is a fiber, that's the wrong
behaviour: blocking the thread also stalls every other fiber and the
outer event loop driving them. The most common way to hit this is a
fiber that creates its own inner event loop (e.g. a server-style fiber
that wants to dispatch its own sources independently of whatever loop
the test or supervising fiber is running on) — with the existing
implementation the inner sd_event_run() would hold the thread while the
outer scheduler should be free to advance other fibers.
Add an event_run_suspend() variant in sd-event/event-future.c that
performs the same prepare/wait/dispatch dance, but when the fast path
finds nothing ready it (a) creates an IO future watching the inner
event loop's epoll fd on the *outer* event loop, (b) optionally creates
a time future for the timeout, and (c) suspends the fiber. When either
future fires the fiber is resumed and the prepare/wait/dispatch sequence
runs once more to actually dispatch what's pending. sd_event_run()
checks sd_fiber_is_running() and delegates to this variant when on a
fiber; profile_delays accounting is intentionally skipped on that path
since the underlying prepare/wait/dispatch primitives already account
for themselves.
Add a family of sd_fiber_*() I/O wrappers that, when called from a
fiber, behave like blocking I/O from the caller's perspective but
yield to the event loop instead of blocking the thread:
sd_fiber_read / sd_fiber_write
sd_fiber_readv / sd_fiber_writev
sd_fiber_recv / sd_fiber_send
sd_fiber_connect
sd_fiber_recvmsg / sd_fiber_sendmsg
sd_fiber_recvfrom / sd_fiber_sendto
sd_fiber_accept
sd_fiber_ppoll
Most of them share a single helper, fiber_io_operation(), which when
invoked outside a fiber falls through to the underlying syscall
directly, preserving the regular blocking behaviour. Inside a fiber
the helper flips the fd to non-blocking (restoring its original mode
on return), tries the syscall once on the fast path, and on EAGAIN/
EWOULDBLOCK creates an sd-event-backed IO future via future_new_io(),
suspends the fiber, and retries the syscall once the event source
fires.
future_new_io() itself is added to sd-event/event-future.{c,h} as a
new IoFuture kind. It wraps sd_event_add_io() into an sd_future:
oneshot enable, EPOLLERR translated via SO_ERROR (suppressed for
non-sockets), and the fd duplicated with F_DUPFD_CLOEXEC to avoid
EEXIST when multiple sources watch the same descriptor.
Together these let fiber-using code write straight-line socket and
pipe I/O without bundling state into callbacks.
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
For libc syscall wrappers (pidfd_open, fsopen, openat2, etc.) we previously
gated the calls behind build-time HAVE_* checks. Replace these with shim
functions in src/libc/ that fall back to the raw syscall at runtime when the
loaded glibc lacks the symbol. The infrastructure lives in src/libc/libc-shim.h:
DEFINE_SYSCALL_SHIM falls back to a direct syscall, DEFINE_LIBC_SHIM returns
ENOSYS (for posix_spawn-family helpers that have no corresponding syscall), and
DEFINE_LIBC_ERRNO_SHIM sets errno=ENOSYS and returns -1 (for read/write-style
helpers). The weak reference to the libc symbol is bound via __asm__(\"name\")
rename so the bare libc identifier never appears as a C token — this avoids
both #undef boilerplate against override-header redirects and the resulting
-Wredundant-decls warning. Drop the corresponding cc.has_function() loop from
meson.build.
For optional libraries (libcryptsetup, libdw, libarchive), drop the per-symbol
HAVE_* checks. Always declare the prototypes, suppressing the redundant-decl
warnings via DISABLE_WARNING_REDUNDANT_DECLS and NOLINT, and resolve the symbols
after the main dlopen via a new DLSYM_OPTIONAL() helper that only assigns on
success. libcryptsetup's crypt_set_keyring_to_link / crypt_token_set_external_path
and libarchive's *_is_set wrappers use fallback functions as their pointer
initializers (returning -ENOSYS and 0 respectively), so call sites can invoke
the symbol unconditionally and just check for -ENOSYS where the \"not supported\"
distinction matters.
The same shim treatment applies to pidfd_spawn / posix_spawnattr_setcgroup_np
(src/libc/spawn.c) and epoll_pwait2 (src/libc/epoll.c), with corresponding
override headers in src/include/override/spawn.h and
src/include/override/sys/epoll.h. posix_spawn_wrapper() in process-util.c and
epoll_wait_usec() in sd-event.c now detect ENOSYS in the return value instead
of checking the function pointer, falling back to plain posix_spawn() and
epoll_wait() respectively. coredump-config and coredump-submit get a
dlopen_dw_has_dwfl_set_sysroot() helper. The kexec arch gate now uses
defined(__NR_kexec_file_load) directly; pidfd.h uses __has_include_next() to
decide whether to pull in glibc's header.
This lets binaries built against newer glibc / libcryptsetup / libdw /
libarchive headers still load and run on older targets where these symbols are
absent.
Coverity flags si.ssi_signo as tainted data from read(), and warns
that casting it to signed could produce a negative value. Add an
explicit range check against INT_MAX before the SIGNAL_VALID check
to prove the cast is safe.
CID#1548033
Follow-up for c8b53fcfd3
Coverity flags the signal_sources array access as a potential
out-of-bounds read because it cannot trace through the SIGNAL_VALID()
macro to know that ssi_signo < _NSIG. Add an explicit assert after
the runtime check to make the constraint visible to static analyzers.
CID#1548033
Follow-up for 7a64c5f23e
Our fuzzer CI recently got bumped to Ubuntu 24.04 with
glibc >= 2.35. Apparently msan is not happy with the new
epoll_pwait2(), hence explicitly mark the memory region
as initialized.
Fixes#40455
Alternative to #40463
We drop the optimization for verify_sigchld as the
check differs based on the options passed and the
extra syscalls are certainly not going to matter
compared to the cost of forking a child process.
Fixes#40166
Now that sd-event doesn't insist on blocking
SIGCHLD anymore when watching for process exit,
let's stop blocking SIGCHLD tree-wide where not
required anymore.
First, don't require blocking SIGCHLD for WEXITED. We watch for WEXITED
via pidfd instead of signalfd, so no need to insist on blocking SIGCHLD
anymore if we're only watching for WEXITED.
Second, do a proper check to see if the kernel autoreaping logic is
enabled. That has nothing to do with SIGCHLD being blocked for the current
thread or not. Instead, the kernel autoreaping logic is enabled either if
the disposition is set to SIG_IGN or if the SA_NOCLDWAIT flag is enabled.
We have dropped cgroup v1 support in v258. Let's drop legacy code.
Then, we can drop 'controller' argument from cg_pid_get_path() and
cg_pidref_get_path().
Defer and exit event sources are marked pending once when they are added
and never again afterwards. This means their pending_iteration is never
incremented after they are initially added, which breaks fairness among
event sources with equal priority which depend on the pending_iteration
variable getting updated in source_set_pending(). To fix this, let's assign
iterations for defer and exit sources in source_dispatch() instead so that
those get their pending_iteration updated as well.
Sometimes it's hard to assign responsibility to a specific event source
for exiting when there's no more work to be done. So let's add exit-on-idle
support where we exit when there are no more event sources.
As documented in sd_event_add_memory_pressure(), we can only add
the memory pressure fd to epoll once we've written the watch string,
so make sure we don't register the memory pressure in
event_source_online() until we've written the watch string.
More post event sources might get added during dispatching, we want
to make sure those become pending as well if we're dispatching a non-post
event source.
Let's not leak details from src/shared and src/libsystemd into
src/basic, even though you can't actually do anything useful with
just forward declarations from src/shared.
The sd-forward.h header is put in src/libsystemd/sd-common as we
don't have a directory for shared internal headers for libsystemd
yet.
Let's also rename forward.h to basic-forward.h to keep things
self-explanatory.
Even when we receive an inotify event, there is no relevant event source
exists. In that case, we need to drop the event from the buffer,
otherwise we cannot escape from the loop.
Fixes#38265.
Follow-up for dbef4dd4f2. Everything that that
commit says is true, but — at least for me — it wasn't obvious why the code is
correct and we can do fixed-size allocations like new(struct inotify_data, 1).
In sd_event_source.child, we have 5 bools. If we make them each take one byte,
the structure size increases. So let's do that for the three of them, and leave
the other two (less frequently used) squished into the last byte. This allows
more efficient code to be generated, without changing the size of the struct:
$ diff -u <(objdump -S build/src/shared/libsystemd-shared-258.so.0|awk '/^static void source_io_unregister/,/^\}/') \
<(objdump -S build/src/shared/libsystemd-shared-258.so|awk '/^static void source_io_unregister/,/^\}/')
s->io.registered = false;
- 34d46f: 48 8b 45 d8 mov -0x28(%rbp),%rax
- 34d473: 0f b6 90 a4 00 00 00 movzbl 0xa4(%rax),%edx
- 34d47a: 83 e2 fe and $0xfffffffe,%edx
- 34d47d: 88 90 a4 00 00 00 mov %dl,0xa4(%rax)
- 34d483: eb 04 jmp 34d489 <source_io_unregister+0x1ca>
+ 34bffe: 48 8b 45 d8 mov -0x28(%rbp),%rax
+ 34c002: c6 80 a4 00 00 00 00 movb $0x0,0xa4(%rax)
+ 34c009: eb 04 jmp 34c00f <source_io_unregister+0x1be>
return;
We use symbols provided by unistd.h without including it. E.g.
open(), close(), read(), write(), access(), symlink(), unlink(), rmdir(),
fsync(), syncfs(), lseek(), ftruncate(), fchown(), dup2(), pipe2(),
getuid(), getgid(), gettid(), getppid(), pipe2(), execv(), _exit(),
environ, STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, F_OK, and their
friends and variants, so on.
Currently, unistd.h is indirectly included mainly in the following two paths:
- through missing_syscall.h, which is planned to covert to .c file.
- through signal.h -> bits/sigstksz.h, which is new since glibc-2.34.
Note, signal.h is included by sd-eevent.h. So, many source files
indirectly include unistd.h if newer glibc is used.
Currently, our baseline on glibc is 2.31. We need to support glibc older
than 2.34, but unfortunately, we do not have any CI environments with
such old glibc. CIFuzz uses glibc-2.31, but it builds only fuzzers, and
many files are even not compiled.
Yu Watanabe
Frantisek Sumsal
Daan De Meyer
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
Michael Vogt
Mike Yuan
Daan De Meyer
Luca Boccassi
Zbigniew Jędrzejewski-Szmek