Use checked arithmetic before clamping the journal iovec length.
Stop copying input iovecs once the fixed journal vector is full.
Follow-up for e69cc9eb36
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
Add the needed assert changes to make the code clean
for the new check-pointer-deref script.
Note that for `_cmp()` functions its a bit of a question
if we want `assert()` or `POINTER_MAY_BE_NULL()` here. The
former is more correct but it adds a (small) runtime hit. If
we use POINTER_MAY_BE_NULL() it is compiled out but strictly
speaking it is wrong (so at least we would need a comment).
Nowadays, we append an extra NUL for each data if possible for safety.
We already do the same for example at write_to_kmsg(), log_do_context(),
write_to_journal(), log_struct_iovec_internal(), and so on.
This does not change any behavior, as the iov_len field is unchanged.
Various functions in log.h are only used by asserts, and there's
enough assertion related stuff in macro.h to justify a separate header
which also makes it easier to avoid circular dependencies.
Let's introduce assert-util.h and an accompanying fundamental header
and move all the assertion related stuff over there. PROJECT_FILE is
moved over to macro.h.
This introduce LOG_ITEM() macro that checks arbitrary formats in
log_struct().
Then, drop _printf_ attribute from log_struct_internal(), as it does not
help so much, and compiler checked only the first format string.
Hopefully, this silences false-positive warnings by Coverity.
We always validate that the target value is below _LOG_TARGET_SINGLE_MAX
before acessing it, but we don't actually size the array like that.
let's fix that.
This doesn#t effectively change anything, but it makes things more
explicit what the limit here is.
We typically want to deal in usec_t, hence let's change the prototype
accordingly, and do proper range checks. Also, make sure are not
confused by negative times.
Do something similar for mktime_or_timegm().
This is a more comprehensive alternative to #34065
Replaces: #34065
All messages logged from exec_spawn() are attributed to the unit
and as such we should set the log level to the unit's max log level
for the duration of the function.
This is a lot of stuff, and sometimes quite wild, let's turn this into
its own header.
All stuff color-related that just generates sequences is now in
ansi-color.h (no .c file!), and everything more complex that
probes/ineracts with terminals remains in termina-util.[ch]
We shouldn't try to use any ANSI escape sequences if TERM=dumb.
Also, the "\r\n" we output can get interpreted as a double newline
(for example by Github Actions), so let's output just "\n" when
TERM=dumb to clean up the CI logs.
If we're already running in a unit with delegation turned on, let's
skip allocation of a scope unit and cgroup subroot. This allows journald
to correctly attribute the logs of all subprocesses spawned by tests such
as test-execute to the test-execute service when the test is running in a service.
For CI in mkosi, I want to configure systemd to log at debug level
to the journal, but not to the console. While we already have max
level settings for journald's forwarding settings, not every log line
goes to the journal, specifically during early boot and when units
are connected directly to the console (think systemd-firstboot), so
let's extend the log level options we already have to allow specifying
a comma separated list of values and lets allow prefixing values with
the log target they apply to to make this possible.
The glibc API is behind the wrapper is called "secure_getenv()", hence
our wrapper really should keep the order too, otherwise things are just
too confusing.
Triggering assert_return() should be a bug in general, and we should
really fix that. But, previously, it is hard to notice such bug, as
it was not critical.
This is for making CI or our testing environment fail if we unexpectedly
trigger assert_return(). So, hopefully we can easily find such bugs.
This should make sure our log lines look nice even if the tty we are
connected to is in raw mode. Normally, it's the TTY's job to turn an NL
we output into a CRNL and interpret it accordingly. However, if the tty
is in "raw" mode it won't do that. Specifically, this is controlled by
the ONLCR flag on the TTY. A TTY might be in raw mode if our "ptyfwd"
logic is used for example, where a 2nd tty is bi-directionally connected
to the primary tty, and duplicate processing is not desired.
Hence, let's just write out the CR on our own. This will make sure that
whenever we output something subsequent output always continues on the
beginning of the next line again, regardless the mode the TTY is in. Of
course, if the TTY is *not* in raw mode, then the extra CR we now
generate is redundant, but it shouldn't hurt either, as it just moves
the cursor to the front of the line even though already is just there.
We only to that if we actually talk to a TTY though, since we don't want
the extra CRs if we are redirected to a pipe or file or so. We are not
on Windows after all.
Fixes: #30155
Even more than with the previous commit, this is not a trivial function
and there's no reason to believe this will actually be inlined nor that
it would be beneficial.
This is preparation for #28891, which adds a bunch more helpers around
"struct iovec", at which point this really deserves its own .c/.h file.
The idea is that we sooner or later can consider "struct iovec" as an
entirely generic mechanism to reference some binary blob, and is the
go-to type for this purpose whenever we need one.
When trying to log, if we fail we try to close the journal FD. If
it is bad, safe_close() will fail and assert, which will try to log,
which will fail, which will try to close the journal FD...
Infinite recursion looks very pretty live in gdb, but let's avoid
that by immediately invalidating the journal FD before closing it.
Follow-up for: #27734
It makes sense to propagate the select log level we maintain also into
glibc, so that any code that uses syslog() directly that ends up in our
processes (libraries and such) are affected by our settings the same way
as we are ourselves.
Let's avoid confusing developers and users when log messages suddenly
stop getting logged to kmsg because of ratelimiting by logging an
additional message if we start ratelimiting log messages to kmsg.
LOG_SET_PREFIX() sets a logging prefix for the current block. The
prefix is prepended to every logging message in the block, followed
by ": ". If a logging prefix is already configured, it is overridden
for the duration of the block, after which it is restored.
A use case for this macro is when we're operating on an image or
directory (using --root or --image). We can use LOG_SET_PREFIX() to
prefix all logging messages with the directory or image that we're
operating on.
Whenever we're going to close all file descriptors, we tend to close
the log and set it into open when needed mode. When this is done with
the logging target set to LOG_TARGET_AUTO, we run into issues because
for every logging call, we'll check if stderr is connected to the
journal to determine where to send the logging message. This check
obviously stops working when we close stderr, so we settle the log
target before we do that so that we keep using the same logging
target even after stderr is closed.
Now that we have reference counting, it's useful to be able to push
single key values onto the log context separately, so that we don't
have to allocate new storage to join the separate string together into
a single field which means we won't be able to reuse a context containing
the same field.
Let's try to optimize against pushing the same fields multiple times
onto the log context. To achieve this we make the log context reference
counted and return an existing context object if it's using the same
fields.
A consequence of this is that we have to make sure attaching/detaching
is coupled to the lifetime of the context object, so we make the attach
and detach functions private for now. If we need independent attach/detach
in the future, we can make that work with some extra complexity but since
we don't need it yet, let's not support it for now.
Luca Boccassi
Daan De Meyer
Yu Watanabe
Michael Vogt
Lennart Poettering
Daan De Meyer
Ivan Kruglov
hanjinpeng
Zbigniew Jędrzejewski-Szmek
Mike Yuan
Joerg Behrmann
Luca Boccassi