Daan De Meyer
8533513aff
Traditionally, asynchronous programming in systemd has been achieved using sd-event along with the asynchronous interfaces of sd-bus and sd-varlink. This works well when the system is reacting to events and all code triggered by those events can run without blocking. In these scenarios, the global Manager object is passed as userdata to the callback, and the callback can use the stack as usual, declaring local state and ensuring proper cleanup via _cleanup_. Control flow structures, such as loops, work as expected, and everything runs smoothly. However, challenges arise when the code needs to perform long-running operations within these callbacks. Since the system cannot block execution within the callback, we can't directly invoke a long-running operation and wait for its result without introducing complexities. Instead, we need to initiate the long-running task, register for completion with sd-event, sd-bus, or sd-varlink, and provide a callback to be invoked when the operation completes. This callback, however, only receives a single userdata pointer, which forces us to bundle all local variables into a struct and pass it along as part of the callback. On top of that, after queuing the asynchronous operation, the caller continues executing. As the caller's stack unwinds when the function exits, the resources and state within the local scope may be prematurely cleaned up. Therefore, the struct must store copies of the local variables or ensure proper reference counting to prevent premature resource cleanup. When multiple long-running operations need to be initiated within a loop, the complexity grows further. We must introduce additional shared state to track the completion of all operations before we can run any code that depends on their results. Furthermore, since the daemon may be shut down at any time, we must track the lifecycle of each long-running operation in the global Manager struct, ensuring proper cleanup even when stack unwinding can no longer manage the resources for us. Fibers, or green threads, provide a more natural way of handling asynchronous operations. By enabling cooperative multitasking within a single thread, fibers allow us to write code that looks like it’s running synchronously, but with the ability to yield control at predefined points, such as when waiting for long-running tasks to complete. With fibers, we can simplify the control flow by running asynchronous operations within a fiber, allowing us to "pause" execution while waiting for the long-running operation to finish and then "resume" the operation once it's complete. This eliminates the need for multiple callback chains, extensive state tracking, and the potential pitfalls of stack unwinding. This commit introduces the ability to execute long-running operations in a non-blocking manner while maintaining the simplicity and readability of synchronous code. The fiber-based approach will significantly improve the handling of complex workflows, making the code easier to write and maintain. The implementation is based on ucontext.h's makecontext() (with a fallback to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp() and sd-event. ucontext.h provides us with alternate stacks that we can switch between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the latter forcibly saves/restores a per context signal mask every time it is called. Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain a per thread signal mask, which makes much more sense than having a per fiber signal mask. The default stack size is the same as a regular thread. Because we use mmap() to allocate the stack, the memory won't actually be used until it is paged in by the kernel, so we don't actually use 8MB per fiber. To integrate fibers with the event loop, each fiber is assigned a deferred event source which resumes the fiber when enabled. The deferred event source is oneshot by default so the fiber will run immediately until it yields or suspends. If it yields, the deferred event source is enabled again (oneshot) immediately. If it suspends, before it suspends, one or more event sources are registered with sd-event that will enable the deferred event source (oneshot) to resume the fiber once the operation it is waiting for completes. Yielding or suspending the fiber is done by calling sd_fiber_yield() or sd_fiber_suspend() respectively. Both of these return zero on success or any error value from the async operation that caused the fiber to resume. This is also how fiber cancellation is implemented. When a fiber is cancelled, sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber is resumed, allowing the fiber to unwind its stack (which allows cleanup to happen automatically) and finish. Instead of having applications work directly with fibers, we hide them behind a generic futures interface to represent long-running operations, regardless of whether those operations are running on a fiber or not. Aside from fibers, the futures library (sd-future) will for example allow waiting for sd-event sources and doing sd-bus calls in the background as well. Fibers can suspend until a future is ready with sd_fiber_await() or by having the future wake up the fiber explicitly in its callback. A future always defaults to waking up the current fiber. Each future kind plugs into the library by providing an sd_future_ops vtable (alloc, free, cancel, set_priority). The library treats the impl pointer returned by alloc() as a black box. Future Implementations retrieve it via sd_future_get_private(). A future starts in SD_FUTURE_PENDING and transitions exactly once to SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that transition either by installing a one-shot callback with sd_future_set_callback() (callback-style code) or by waiting on it from a fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await() is itself built on a "wait future" that resolves when its target resolves; sd_future_new_wait() exposes the same primitive directly so non-fiber callers can chain futures without involving a fiber. Cancellation is cooperative: sd_future_cancel() invokes the future impl's cancel callback, which is responsible for tearing down its work and ultimately resolving the promise with -ECANCELED. For fiber futures this is what surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend() mentioned above. Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() — take a self-reference on their future so they outlive the caller's scope. The self-ref is dropped when the fiber resolves. This floating mechanism (sd_fiber_set_floating()) is restricted to fiber futures because they uniquely guarantee resolution; allowing it for arbitrary future kinds would risk silent leaks for kinds that may never resolve. Note that fiber cleanup depends on the runtime operating normally. Each fiber's _cleanup_-style cleanups live on the fiber's own stack and run only when the fiber is resumed and allowed to unwind, which requires a working event loop to drive it to completion. The exit event source registered for top-level fibers ensures unwind on a normal sd_event_exit(), but if the event loop itself terminates abnormally (e.g. an unrecoverable allocation failure mid-dispatch) before all fibers have resolved, their stacks never unwind and any resources they own leak. The code lives in libsystemd as sd-future (not exported) for the following reasons: - We may want to make this a public libsystemd API in the future - The code can't live in src/basic as it makes heavy use of sd-event - The code can't live in src/shared as sd-bus and sd-event make use of it The log and log-context headers are updated with functions to allow fibers to have their own log prefix and log context.
Integration tests
Running the integration tests with meson + mkosi
To run the integration tests with meson + mkosi, make sure you're running the
latest version of mkosi. See
docs/HACKING.md
for more specific details. Make sure mkosi is available in $PATH when
reconfiguring meson to make sure it is picked up properly.
If you haven't done it already, be sure to generate signing keys for mkosi:
$ mkosi genkey
Next, we can build the integration test image with meson:
$ mkosi -f box -- meson compile -C build mkosi
By default, the mkosi meson target which builds the integration test image depends on
other meson targets to build various systemd tools that are used to build the image to make
sure they are up-to-date. If you instead want the already installed systemd tools on the
host to be used, you can run mkosi manually to build the image. To build the integration test
image without meson, run the following:
$ mkosi -f
Note that by default we assume that build/ is used as the meson build directory that will be used to run
the integration tests. If you want to use another directory as the meson build directory, you will have to
configure the mkosi build directory (BuildDirectory=), cache directory (CacheDirectory=) and output
directory (OutputDirectory=) to point to the other directory using mkosi/mkosi.local.conf.
After the image has been built, the integration tests can be run with:
$ mkosi -f box -- meson test -C build --setup=integration --suite integration-tests --num-processes "$(($(nproc) / 4))"
As usual, specific tests can be run in meson by appending the name of the test which is usually the name of the directory e.g.
$ mkosi -f box -- meson test -C build --setup=integration -v TEST-01-BASIC
See mkosi -f box -- meson introspect build --tests for a list of tests.
To interactively debug a failing integration test, the --interactive option
(-i) for meson test can be used. Note that this requires meson v1.5.0 or
newer:
$ mkosi -f box -- meson test -C build --setup=integration -i TEST-01-BASIC
Due to limitations in meson, the integration tests do not yet depend on the mkosi target, which means the mkosi target has to be manually rebuilt before running the integration tests. To rebuild the image and rerun a test, the following command can be used:
$ mkosi -f box -- meson compile -C build mkosi && mkosi -f box -- meson test -C build --setup=integration -v TEST-01-BASIC
The integration tests use the same mkosi configuration that's used when you run
mkosi in the systemd reposistory, so any local modifications to the mkosi
configuration (e.g. in mkosi/mkosi.local.conf) are automatically picked up and used
by the integration tests as well.
Iterating on an integration test
To iterate on an integration test, let's first get a shell in the integration test environment by running the following:
$ mkosi -f box -- meson compile -C build mkosi && mkosi -f box -- meson test -C build --setup=shell -i TEST-01-BASIC
This will get us a shell in the integration test environment after booting the machine without running the
integration test itself. After booting, we can verify the integration test passes by running it manually,
for example with systemctl start TEST-01-BASIC.
Now you can extend the test in whatever way you like to add more coverage of existing features or to add coverage for a new feature. Once you've finished writing the logic and want to rerun the test, run the the following on the host:
$ mkosi -R
This will rebuild the distribution packages without rebuilding the entire integration test image. Next, run the following in the integration test machine:
$ systemctl soft-reboot
$ systemctl start TEST-01-BASIC
A soft-reboot is required to make sure all the leftover state from the previous run of the test is cleaned
up by soft-rebooting into the btrfs snapshot we made before running the test. After the soft-reboot,
re-running the test will first install the new packages we just built, make a new snapshot and finally run
the test again. You can keep running the loop of mkosi -R, systemctl soft-reboot and
systemctl start ... until the changes to the integration test are working.
If you're debugging a failing integration test (running meson test --interactive),
there's no need to run systemctl start ..., running systemctl soft-reboot on its own is sufficient to
rerun the test.
Configuration variables
TEST_NO_QEMU=1: Don't run tests under qemu.
TEST_PREFER_QEMU=1: Run all tests under qemu.
TEST_NO_KVM=1: Disable qemu KVM auto-detection (may be necessary when you're
trying to run the vanilla qemu and have both qemu and qemu-kvm installed)
TEST_MATCH_SUBTEST=subtest: If the test makes use of run_subtests use this
variable to provide a POSIX extended regex to run only subtests matching the
expression.
TEST_MATCH_TESTCASE=testcase: Same as $TEST_MATCH_SUBTEST but for subtests
that make use of run_testcases.
TEST_SKIP: takes a space separated list of tests to skip.
TEST_SKIP_SUBTEST=subtest: takes a space separated list of subtests to skip.
TEST_SKIP_TESTCASE=testcase: takes a space separated list of testcases to skip.
TEST_SAVE_JOURNAL=0|1|fail: When 0, journal file will be removed on exit.
When 1, journal file will be saved at $BUILD_DIR/test/journal. When fail,
journal file will be saved only when the test is failed. Defaults to fail.
TEST_JOURNAL_USE_TMP=1: Write test journal to /tmp while the test is in
progress and only move the journal to its final location in the build directory
($BUILD_DIR/test/journal) when the test is finished.
Running the integration tests without building systemd from source
If you want to run the integration tests against prebuilt systemd packages,
first add the following to mkosi/mkosi.local.conf to stop mkosi from building
systemd packages from source:
[Build]
Environment=NO_BUILD=1
You'll then probably want to use the PackageDirectories= or SandboxTrees=
options to provide mkosi with a directory containing the systemd packages or a
repository file that points to a repository with the systemd packages that
should be installed.
If the rpms are installed on the host system that the tests are running on,
you'll probably want to disable usage of the tools tree so that the tools from
the host system are used to build the image by adding the following to
mkosi/mkosi.local.conf:
[Build]
ToolsTree=
On the other hand, if the rpms are available but not installed on the host
system, you'll want to make sure they're installed into the mkosi tools tree so
that they're used to build the image by again using either
PackageDirectories= or SandboxTrees=, but this time in
mkosi/mkosi.tools.conf/mkosi.local.conf instead of mkosi/mkosi.local.conf.
Finally, we'll make use of the standalone mode of running the integration tests to avoid having to install any build dependencies.
$ mkosi -f box -- meson setup testsuite test/integration-tests/standalone
$ mkosi -f
$ mkosi box -- meson test -C testsuite --num-processes "$(($(nproc) / 4))"
SELinux AVCs
To have TEST-06-SELINUX check for SELinux denials, write the following to
mkosi/mkosi.local.conf:
[Runtime]
KernelCommandLineExtra=systemd.setenv=TEST_SELINUX_CHECK_AVCS=1
Ubuntu CI
New PRs submitted to the project are run through regression tests, and one set of those is the 'autopkgtest' runs for several different architectures, called 'Ubuntu CI'. Part of that testing is to run all these tests.
Known issues affecting the infrastructure/testbed can be seen on this Ubuntu page:
https://discourse.ubuntu.com/t/autopkgtest-service/34490
In case a test fails, the full set of artifacts, including the journal of the failed run, can be downloaded from the artifacts.tar.gz archive which will be reachable in the same URL parent directory as the logs.gz that gets linked on the Github CI status.
The log URL can be derived following a simple algorithm, however the test completion timestamp is needed and it's not easy to find without access to the log itself. For example, a noble s390x job started on 2024-03-23 at 02:09:11 will be stored at the following URL:
Fortunately a list of URLs listing file paths for recently completed test runs is available at:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble-upstream-systemd-ci-systemd-ci/
paths listed at this URL can be appended to the URL to download them. Unfortunately there are too many results and the web server cannot list them all at once. Fortunately there is a workaround: copy the last line on the page, and append it to the URL, with a '?marker=' prefix, and the web server will show the next page of results. For example:
The 5 characters at the end of the last directory are not random, but the first 5 characters of a SHA1 hash generated based on the set of parameters given to the build plus the completion timestamp, such as:
$ echo -n 'systemd-upstream {"build-git": "https://salsa.debian.org/systemd-team/systemd.git#debian/master", "env": ["UPSTREAM_REPO=https://github.com/systemd/systemd.git", "CFLAGS=-O0", "DEB_BUILD_PROFILES=pkg.systemd.upstream noudeb", "TEST_UPSTREAM=1", "CONFFLAGS_UPSTREAM=--werror -Dslow-tests=true", "UPSTREAM_PULL_REQUEST=31444", "GITHUB_STATUSES_URL=https://api.github.com/repos/systemd/systemd/statuses/c27f600a1c47f10b22964eaedfb5e9f0d4279cd9"], "ppas": ["upstream-systemd-ci/systemd-ci"], "submit-time": "2024-02-27 17:06:27", "uuid": "02cd262f-af22-4f82-ac91-53fa5a9e7811"}' | sha1sum | cut -c1-5
To add new dependencies or new binaries to the packages used during the tests, a merge request can be sent to: https://salsa.debian.org/systemd-team/systemd targeting the 'upstream-ci' branch.
The cloud-side infrastructure, that is hooked into the Github interface, is located at:
https://git.launchpad.net/autopkgtest-cloud/
A generic description of the testing infrastructure can be found at:
https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure
In case of infrastructure issues with this CI, things might go wrong in two places:
- starting a job: this is done via a Github webhook, so check if the HTTP POST are failing on https://github.com/systemd/systemd/settings/hooks
- running a job: all currently running jobs are listed at https://autopkgtest.ubuntu.com/running#pkg-systemd-upstream in case the PR does not show the status for some reason
- reporting the job result: this is done on Canonical's cloud infrastructure, if jobs are started and running but no status is visible on the PR, then it is likely that reporting back is not working
The CI job needs a PPA in order to be accepted, and the upstream-systemd-ci/systemd-ci PPA is used. Note that this is necessary even when there are no packages to backport, but by default a PPA won't have a repository for a release if there are no packages built for it. To work around this problem, when a new empty release is needed the mark-suite-dirty tool from the https://git.launchpad.net/ubuntu-archive-tools can be used to force the PPA to publish an empty repository, for example:
$ ./mark-suite-dirty -A ppa:upstream-systemd-ci/ubuntu/systemd-ci -s noble
will create an empty 'noble' repository that can be used for 'noble' CI jobs.
For infrastructure help (e.g.: Github token refresh) a bug can be filed at: https://launchpad.net/auto-package-testing or an email can be sent to the ubuntu-quality mailing list: https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality
Given access to the shared secret, tests can be re-run using the generic retry-github-test tool:
A wrapper script that makes it easier to use is also available:
https://piware.de/gitweb/?p=bin.git;a=blob;f=retry-gh-systemd-Test
Manually running a part of the Ubuntu CI test suite
In some situations one may want/need to run one of the tests run by Ubuntu CI locally for debugging purposes. For this, you need a machine (or a VM) with the same Ubuntu release as is used by Ubuntu CI (Jammy ATTOW).
First of all, clone the Debian systemd repository and sync it with the code of
the PR (set by the $UPSTREAM_PULL_REQUEST env variable) you'd like to debug:
$ git clone https://salsa.debian.org/systemd-team/systemd.git
$ cd systemd
$ git checkout ci/v<XYZ>-stable
$ TEST_UPSTREAM=1 UPSTREAM_PULL_REQUEST=12345 ./debian/extra/checkout-upstream
Now install necessary build & test dependencies:
# PPA with some newer Ubuntu packages required by upstream systemd
$ add-apt-repository -y --enable-source ppa:upstream-systemd-ci/systemd-ci
$ apt build-dep -y systemd
$ apt install -y autopkgtest fakemachine qemu-system-x86
Build systemd deb packages with debug info:
$ TEST_UPSTREAM=1 DEB_BUILD_OPTIONS="nocheck nostrip noopt pkg.systemd.upstream" dpkg-buildpackage -us -uc -b
$ cd ..
Prepare a testbed image for autopkgtest (tweak the release as necessary):
$ autopkgtest-buildvm-ubuntu-cloud --ram-size 1024 -v -a amd64 -r noble
And finally run the autopkgtest itself:
$ autopkgtest -o logs *.deb systemd/ \
--env=TEST_UPSTREAM=1 \
--timeout-factor=3 \
--test-name=boot-and-services \
--shell-fail \
-- autopkgtest-virt-qemu --cpus 4 --ram-size 2048 autopkgtest-noble-amd64.img
where --test-name= is the name of the test you want to run/debug. The
--shell-fail option will pause the execution in case the test fails and shows
you the information how to connect to the testbed for further debugging.
Running Coverity locally
Note: this requires a Coverity license, as the public tool tarball doesn't contain cov-analyze and friends, so the usefulness of this guide is somewhat limited.
Debugging certain pesky Coverity defects can be painful, especially since the OSS Coverity instance has a very strict limit on how many builds we can send it per day/week, so if you have an access to a non-OSS Coverity license, knowing how to debug defects locally might come in handy.
After installing the necessary tooling we need to populate the emit DB first:
$ rm -rf build cov
$ meson setup build -Dman=false
$ cov-build --dir=./cov ninja -C build
From there it depends if you're interested in a specific defect or all of them. For the latter run:
$ cov-analyze --dir=./cov --wait-for-license
If you want to debug a specific defect, telling that to cov-analyze speeds things up a bit:
$ cov-analyze --dir=./cov --wait-for-license --disable-default --enable ASSERT_SIDE_EFFECT
The final step is getting the actual report which can be generated in multiple formats, for example:
$ cov-format-errors --dir ./cov --text-output-style multiline
$ cov-format-errors --dir=./cov --emacs-style
$ cov-format-errors --dir=./cov --html-output html-out
Which generate a text report, an emacs-compatible text report, and an HTML report respectively.
Other useful options for cov-format-error include --file <file> to filter out
defects for a specific file, --checker-regex DEFECT_TYPE to filter our only a
specific defect (if this wasn't done already by cov-analyze), and many others,
see --help for an exhaustive list.
Code coverage
We have a daily cron job in Github Actions which runs all unit and integration tests, collects coverage using gcov/lcov, and uploads the report to Coveralls. In order to collect the most accurate coverage information, some measures have to be taken regarding sandboxing, namely:
- ProtectSystem= and ProtectHome= need to be turned off
- the coverage files (*.gcda) files need to be present in the image and need to be writable by all processes
The first point is relatively easy to handle and is handled automagically by
mkosi by creating the necessary dropins when COVERAGE=1 is passed via the
Environment= setting.
Making the coverage files accessible and writable to everything is achieved by
pre-creating all the files and making them world readable and writable. However,
this is not enough in some cases, like for services that use DynamicUser=yes,
since that implies ProtectSystem=strict that can't be turned off. A solution to
this is to use ReadWritePaths=/coverage, which works for the majority of
cases, but can't be turned on globally, since ReadWritePaths= creates its own
mount namespace which might break some services. Hence, the
ReadWritePaths=/coverage is enabled for all services with the test- prefix
(i.e. test-foo.service or test-foo-bar.service), both in the system and the user
managers.
So, if you're considering writing an integration test that makes use of
DynamicUser=yes, or other sandboxing stuff that implies it, please prefix the
test unit (be it a static one or a transient one created via systemd-run), with
test-, unless the test unit needs to be able to install mount points in the
main mount namespace - in that case use IGNORE_MISSING_COVERAGE=yes in the
test definition (i.e. TEST-*-NAME/test.sh), which will skip the post-test
check for missing coverage for the respective test.
Fuzzers
systemd includes fuzzers in src/fuzz/ that use libFuzzer and are automatically
run by OSS-Fuzz with sanitizers. To add a
fuzz target, create a new src/fuzz/fuzz-foo.c file with a
LLVMFuzzerTestOneInput function and add it to the list in
src/fuzz/meson.build.
Whenever possible, a seed corpus and a dictionary should also be added with new
fuzz targets. The dictionary should be named src/fuzz/fuzz-foo.dict and the
seed corpus should be built and exported as $OUT/fuzz-foo_seed_corpus.zip in
tools/oss-fuzz.sh.
The fuzzers can be built locally by running tools/oss-fuzz.sh, or by running:
CC=clang CXX=clang++ \
meson setup build-libfuzz -Dllvm-fuzz=true -Db_sanitize=address,undefined -Db_lundef=false \
-Dc_args='-fno-omit-frame-pointer -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION'
ninja -C build-libfuzz fuzzers
Each fuzzer then can be then run manually together with a directory containing the initial corpus:
export UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1
build-libfuzz/fuzz-varlink-idl test/fuzz/fuzz-varlink-idl/
Note: the halt_on_error=1 UBSan option is especially important, otherwise the
fuzzer won't crash when undefined behavior is triggered.
You should also confirm that the fuzzers can be built and run using the OSS-Fuzz toolchain:
path_to_systemd=...
git clone --depth=1 https://github.com/google/oss-fuzz
cd oss-fuzz
for sanitizer in address undefined memory; do
for engine in libfuzzer afl honggfuzz; do
./infra/helper.py build_fuzzers --sanitizer "$sanitizer" --engine "$engine" \
--clean systemd "$path_to_systemd"
./infra/helper.py check_build --sanitizer "$sanitizer" --engine "$engine" \
-e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd
done
done
./infra/helper.py build_fuzzers --clean --architecture i386 systemd "$path_to_systemd"
./infra/helper.py check_build --architecture i386 -e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd
./infra/helper.py build_fuzzers --clean --sanitizer coverage systemd "$path_to_systemd"
./infra/helper.py coverage --no-corpus-download systemd
If you find a bug that impacts the security of systemd, please follow the guidance in CONTRIBUTING.md on how to report a security vulnerability.
For more details on building fuzzers and integrating with OSS-Fuzz, visit: