The hook that enforces the review deadline used bash's [[ test, but hook
commands run under /bin/sh, which is dash on the runners. Every tool call was
denied with "[[: not found", so all six lenses returned empty findings and
the run reported a clean review in 73 seconds. Use a POSIX test.
An empty result from an agent that never got a tool call through looks exactly
like a clean review, so require at least one successful tool call before a lens
counts as completed. A run like that now fails every lens, which also stops it
from raising the review count.
The "Pull Request Labeler" workflow read the full label set of a pull
request and wrote it back with setLabels(), which is a full replace. Other
workflows label the same pull request at the same time: claude-review.yml
adds the claude-review label on the pull_request_target "opened" event,
which is the very same event that starts the labeler. Whenever that add
landed between the labeler's listLabelsOnIssue() and its setLabels(), the
label was dropped again and the review never ran.
Only ever add or remove the individual labels the labeler owns, so
concurrent changes by other workflows survive. removeLabel() returns 404
when the label is already gone, which is exactly what a racing run causes,
so swallow that status.
sd_fiber_await() returned the target future's result on completion and an error
on interruption, so a negative value could mean either that the target future
resolved with an error or that the wait never finished. Most callers had to check
the target future's state afterwards to tell the two apart. Even worse, a target
resolving with -ECANCELED was indistinguishable from a cancelled wait.
Let's return 0 once the target future has resolved and a negative error only when
the wait itself did not complete. The target future's outcome stays available through
sd_future_result(), which callers have to additionally inspect now.
Reviewers ignored the tool call budget in their prompt, and the budget was a
poor stand-in for what matters anyway: per-call latency varied from 25 to
85 seconds between lenses, so the same count meant 7 minutes for one and 20
for another. Two lenses hit the timeout and lost everything they had found.
Enforce the deadline itself. run_claude exports it per invocation and a
PreToolUse hook denies further tool calls once it has passed, telling the
model to return what it has. The structured output call stays allowed, so
a slow lens still produces findings. The timeout, three minutes later, only
catches a hang. Give reviewers 22 minutes and the validator 12.
AI agents prepend a notice to README.md that the author removes once they
have reviewed the changes. Until that happens nobody else should look at
the PR: convert it to a draft, don't add the please-review label, and
don't start a Claude review. The labeler and the review workflow check
README.md at the PR head for the notice. Marking the PR ready for review
after removing the notice triggers the normal labeling again.
We replace the implicit single callback and wait-future indirection with
the usual slot design so independent fibers and future groups can track
the same future. We also opt to always have callbacks dispatch asynchronously,
including when they are registered after the future has already resolved.
We now require futures to be resolved before their final unref rather than
cancelling them implicitly. The idea is to catch bugs rather than leak futures.
A future should always be cancelled and awaited before it is cleaned up,
otherwise resources might leak if the stack is not unwound properly. We make
exceptions for futures that haven't started executing yet as there's no risk
of leaking resources there since there's no stack to unwind.
One issue with awaiting futures is that sd_fiber_await() returned both
errors and the future's result via the function's return value, meaning it
was impossible to distinguish future failures from interrupted waits (or
waits that failed for whatever other reason). This is generally not an issue
except in the cleanup functions, where we have to retry on interrupted waits
(the best example is a parent fiber waiting for a future being cancelled
itself, which should cancel the future it's waiting for and retry the wait).
So we introduce fiber_wait() which distinguishes between both values via
return value and output argument so the two can be distinguished from each
other.
Cleanup always consumes the caller's reference, including on invalid self or
cross-loop waits. The existing assertion still forbids dropping the last
reference to an unresolved future. The installed header documents this rule.
We introduce a new RUNNING fiber state to distinguish executing fibers from
non-executing fibers. We replace CANCELLED with queued resume results and
preserve interruptions across ordinary wakeups.
We add future groups with wait-all, wait-any and ignore-errors policies to
wait for multiple futures at once and migrate various existing call sites over
to them. Group size is a direct size_t query, treating NULL as an empty group.
Document all four policy combinations, including that IGNORE_ERRORS suppresses
fail-fast and parent cancellation without discarding the group's error result.
We also add defer futures and migrate bus, varlink, QMP and fiber I/O callers to
use them. Bus and QMP futures report -ENOPKG when no event loop is attached,
rather than passing NULL to sd_future_new() and returning -EINVAL.
Parent-await suppression is derived from the active wait's callback slot, so
groups use sd_fiber_await() without a separate await API or additional state.
Waits, group children and captured parents must use the same event loop.
Groups created on another loop have no parent, and a group releases its
parent reference when that fiber finishes.
Resolve SD_EVENT_DEFAULT in sd_future_new() so futures retain the actual
loop. Match group parents against that resolved loop as well; an unavailable
default returns -ENOPKG without creating an event loop implicitly.
Keep the public await entry point in fiber.c, sharing the slot-based wait with
cancellation cleanup through future-internal.h. Keep event/future integration
in event-future.c/h: priority inheritance uses the public fiber accessor, and
sd_future_new_defer() lives alongside the other event-backed constructors.
This avoids sd-event depending on sd-future private headers and removes the
redundant private defer wrapper.
Assert the non-failing slot-disabling and active-loop invariants instead of
returning early without completing callback dispatch.
Regression tests cover group error propagation and NULL-safe size queries,
interrupted waits, partial add-many rollback, groups outliving their parent,
and cancellation or timeout during nested event-loop waits, including
event-loop and descriptor cleanup. They also cover cross-loop parent isolation,
reference release on invalid cleanup calls, mixed WAIT_ANY outcomes, failed
child cancellation and policy changes on resolved groups. Additional tests
verify failed group-priority updates, slot priority during dispatch and exit,
and cancellation, timeout and spurious wakeups during bus and QMP calls. Late
replies must leave subsequent calls unaffected. Default-event tests cover
missing-loop errors, reference lifetime and parent capture by constructors.
Unattached bus and QMP clients are covered by missing-event-loop regressions.
Constructor tests verify inherited and default priorities for I/O, defer and
timer futures, plus defer argument validation and missing-default errors.
Signed-off-by: Daan De Meyer <daan@amutable.com>
A triple backtick inside prose made the fence count odd, so a body with an
inline marker plus a valid suggestion block got cut at the block's closing
fence and left it open. Only fences at the start of a line delimit blocks,
so count and strip those alone.
Reviews keep hitting the 50 minute timeout. The coordinator agent that was
supposed to launch reviewers, watch the clock, stop stragglers, validate and
write the result does none of that reliably: reviewers get stopped and lose
their findings, foreground validators can't be stopped, deadlines get
overshot by minutes, and one leftover agent keeps the process alive until
the job fails. Piling more rules into its prompt hasn't helped.
Do the orchestration in the shell instead. Run one bounded claude invocation
per review lens in parallel under timeout(1), each returning its findings as
structured output, merge them with jq and run one bounded validator over the
candidates. A lens that times out is just reported as failed. The post job
builds the tracking comment from what it actually posted, so the model no
longer authors the summary or the review bookkeeping. Agents run from an
empty directory with all inputs outside it, so the sandbox keeps both the
checkout and the inputs read-only.
namespace_enter() tried to become root after joining a user namespace
but ignored failures, so if root wasn't mapped we'd just carry on with
our original UID/GID. Let's make that an error instead.
Dropping supplementary groups stays best effort, as setgroups() might
be denied (e.g. by the nsresourced BPF LSM) even when root is mapped.
Some callers do need to enter namespaces without root, so add
NAMESPACE_ENTER_KEEP_UID_GID_IF_ROOT_UNMAPPED for those. It's used by
systemd-analyze unit-shell and by the credential refresh in the user
manager. Everything else enters containers or namespaces we allocated
ourselves, where root is always mapped.
The Claude review workflow posted long, padded inline comments, and every
force push produced a fresh batch of new findings on code that had not
changed. Both make authors stop reading the reviews.
Add a comment style section to the prompt that limits each comment to the
problem and the obvious fix, and truncate anything over 1000 characters in
the post job. Track the number of completed reviews and the last reviewed
head in a hidden state line in the tracking comment. Each successful review
narrows the severities that may be posted: everything on the first review,
no nits on the second, and only must-fix findings from the third on. The
post job enforces the policy so it holds even if the model ignores the
prompt, and reviewers are told to range-diff against the previous head and
skip non-must-fix findings on unchanged code. Commenting "@claude review
full" resets the count for that run.
This adds a fallback to `blockdev_get_root`, to handle the case where
`/usr` is a sysext overlay. Additinal changes are:
- move `extension_overlay_block` to `extension-util.h`, and make it
generic to work in both confext and sysext cases (though only sysext is
actually used currently)
- add a usr-only integration test to exercise the failure
Closes#43565.
The runtime directory holds what the helpers we fork off need: swtpm's socket
and, unless --tpm-state= points elsewhere, its state, the virtiofsd sockets,
the varlink control socket, and the QEMU config file. runtime_dir was declared
after the CLEANUP_ARRAY() that terminates those helpers, and cleanup handlers
run in reverse declaration order, so the directory was removed while all of
them were still running. That produces spurious errors from the helpers, and
anything they create while rm_rf() walks the tree can make the final rmdir fail
and leave the runtime directory behind.
Declare runtime_dir before the CLEANUP_ARRAY() instead, so the helpers are sent
SIGTERM and reaped first and the directory only goes away once nothing writes
to it anymore.
While at it, drop the rm_rf_physical_and_freep() destructors from the SSH key
paths. They are only ever assigned paths below the runtime directory, whose
removal takes them along, so those destructors never had an effect.
Running the review step from /tmp/claude-review-sandbox put the directory the
review result has to be written to outside the agent's working directory. The
Write tool refuses to touch anything outside that directory, no matter what the
permission rules say, so every attempt to write review-result.json was denied.
Runs only passed because the agent fell back to writing the file from a
sandboxed shell command instead, and the last one hit the 50 minute step
timeout before it got that far, leaving nothing for the post job to upload.
The separate working directory bought little on top of the write allow rule,
which already confines the agent to claude-review-work/, so drop it and run in
the workspace again.
Provide a Tokio-like #[tokio::main] entry point for binaries that want to run
their main function inside a fiber on a default sd_event loop.
DEFINE_MAIN_FUNCTION_FIBER(impl) mirrors DEFINE_MAIN_FUNCTION but routes impl
through a new run_main_fiber() helper. impl keeps the run() convention while
the loop's exit code keeps the exit status convention: impl's result never
becomes an exit code, an explicit exit status wins over a successful impl, and
impl's own failure wins over the exit status unless it is the -ECANCELED of a
fiber that the exit request cancelled. Signal policy stays with impl so
existing handlers such as sd_event_set_signal_exit() keep working.
The fiber is created fire-and-forget: as long as the loop reaches
SD_EVENT_FINISHED it unwinds the fiber through its exit event source, so we
never hold a reference we cannot drop because the fiber is still suspended. A
loop that fails internally never dispatches its exit sources, so the fiber may
stay suspended with no working loop left to unwind it; we leak both for now and
leave a TODO to keep the broken loop out of the default slot. Since the fiber
also refers to run_main_fiber()'s stack frame, refuse to run on a loop we don't
own, or from a fiber, as sd_fiber_new() leaves the exit event source disabled
there and nothing would unwind the fiber.
Cover argument forwarding, result propagation, explicit event exits, nested
invocations, signal-source registration, event-loop errors, and fiber stack
unwinding, and check after every run that the default event loop was released
again. Also build and run a separate test executable that expands
DEFINE_MAIN_FUNCTION_FIBER itself and verifies that a positive implementation
result maps to a successful process exit.
Replace opaque built-in review behavior with explicit parallel review lenses for
correctness, concurrency, security, architecture, coding style, and regression
coverage, plus bounded domain-specific reviewers and independent validation.
Prefetch pull request discussion in the setup job while letting the read-only
review job fetch and inspect the full commit history without per-commit
worktrees.
Keep GitHub writes isolated to the setup and posting jobs, pin the Claude models
and CLI, and bound execution below the AWS credential lifetime. Materialize
review instructions from the trusted base revision and scope agent writes to the
structured result directory so untrusted pull request content cannot widen the
review authority.
Treat the posting job as the authoritative trust boundary: validate all
model-selected commits, paths, lines, and thread IDs, normalize degenerate line
ranges, bound sanitized output, and surface GitHub API failures in the tracking
comment before performing writes. Retain daandemeyer/systemd in the repository
allowlist so the workflow can be exercised end to end on the personal fork.
Manual runs against fork PR #15 covered OIDC authentication, sandboxing, result
artifact handoff, tracking-comment updates, and validation of posted output.
Signed-off-by: Daan De Meyer <daan@amutable.com>
The managed-namespace state lives only in the pinned BPF maps, which
survive a plain restart. But an upgrade from a version whose map schema
differs replaces those maps (they are pinned under a new directory), so
they come up empty, and nothing repopulated them: every user namespace that
was already running would silently lose its write restriction (and, for
"self" allocations, its setgroups() denial) until it exited.
Re-add every live registry entry to the BPF maps during manager_startup(),
in the same pass that reaps the dead ones. userns_managed is reconstructed
for all entries; the setgroups() denial is reconstructed from a flag now
persisted in the registry, since it applies only to "self" allocations,
which the map alone cannot tell apart. The update is idempotent when the
pinned maps already carry the entries (the common restart case), and
load-bearing after an upgrade that replaced them with empty ones.
Ordering matters here: the programs are loaded but left unattached until
the maps have been repopulated, and only then does the previous version's
pin directory get removed, which is what detaches its programs. So the old
policy keeps enforcing right up to the moment the new one takes over, and
the new programs never run against a managed set that is still empty. Doing
it the other way around would leave every pre-upgrade namespace failing
open for the duration of the registry sweep, which is long enough for a
client looping over create() to land a transient owned inode on a
persistent file system.
Entries written before that flag existed are exactly the ones the first
such upgrade finds, so they get no answer from it. Recognize those by what
the flag mirrors: a "self" allocation maps the UID of the caller rather
than a transient range, i.e. it is the single UID entry whose start UID is
its owner.
This does not help a downgrade to a version without this change: such a
version cannot clean up the newer pin directory it knows nothing about, so
those programs stay attached until the next reboot clears the tmpfs-backed
bpffs, with enforcement reverting to the older policy in the meantime.
A write through an overlay mount does not create the object on the overlay
itself: overlayfs creates it on the upper layer via vfs_create(), which
reaches only the *inode* LSM hooks, not the *path* hooks the transient
range policy attaches to. The one path hook that does fire sees the overlay
mount, whose superblock belongs to the (managed) mounting namespace and so
looks like it dies with it -- while the upper may be a persistent file
system on which the transient IDs then persist.
Regular file system -- caught by the path hook:
openat(dir_on_persistent_fs, "f", O_CREAT)
path_openat -> ... -> security_path_mknod(path{mnt = persistent mount}) << path hook (attached)
userns_restrict_path_mknod -> validate_mount()
sb->s_user_ns = init -> not below managed (outlives the ns)
mnt_idmap does not remap the transient id (reaches disk)
=> -EPERM BLOCKED
Overlay write -- slips past the path hook (this is the hole):
openat(overlay_mnt/"f", O_CREAT)
path_openat -> ... -> security_path_mknod(path{mnt = OVERLAY mount}) << path hook (attached)
validate_mount(): overlay sb->s_user_ns == managed
=> "dies with the namespace" ALLOW
ovl_create_or_link -> vfs_create(UPPER dentry)
security_inode_create(UPPER inode) << inode hook (NOT attached)
=> no check
inode created on the UPPER fs, owned by the transient uid PERSISTS
Judge the upper at mount time instead, in lsm/sb_kern_mount:
mount("overlay", ..., upperdir=<X>, workdir=<Y>)
do_new_mount -> vfs_get_tree
ovl_fill_super
stamps work/index onto the upper via vfs_mkdir, owned by the
transient mounter -> security_inode_mkdir only (not a path hook)
security_sb_kern_mount(sb) << hook (attached)
userns_restrict_sb_kern_mount:
sb->s_magic != OVERLAYFS_SUPER_MAGIC -> return 0 (not an overlay)
find_managed_userns(sb->s_user_ns) == none -> return 0 (unmanaged mounter)
upper = ((struct ovl_fs*)sb->s_fs_info)->layers[0].mnt
upper == NULL -> return 0 (read-only, no upper)
fs_userns = upper->mnt_sb->s_user_ns
upper dies with the managed ns (e.g. a tmpfs) => return 0 ALLOW
upper persistent, idmap records no transient id => return 0 ALLOW
upper persistent, an idmap extent records a transient => -EPERM REFUSE
The upper has to be judged at mount time because overlayfs performs the
actual upper creations (both the mount-time work/index dirs and every later
copy-up) with internal vfs_create()/vfs_mkdir() calls that reach only the
inode hooks, on the upper's own superblock, where neither the overlay's
ovl_fs nor the mount idmapping is available. Refusing the mount stops every
such write. It cannot undo the work/index dirs ovl_fill_super() already
stamped onto the upper before this hook runs. Reaching those takes an
upperdir and a workdir that already exist on the same persistent,
non-idmapped file system and that the transient IDs may write to: the client
cannot create them itself, since that is what the path hooks deny.
The idmap check walks the upper mount's uid_map and gid_map and permits the
overlay only when no extent records a transient id on disk (ids the idmap
does not map fail the write, so they are fine too). This is what lets a
client back a writable overlay with a persistent but id-mapped upper, e.g.
one handed out by mountfsd, while a plain persistent upper stays refused.
The check keys on whether a writable upper exists (layers[0].mnt is
non-NULL), which is fixed for the superblock's lifetime, not on the mutable
SB_RDONLY flag: an overlay mounted read-only but with an upper can be
remounted read-write later without re-entering sb_kern_mount (remount uses a
different LSM hook), so it must be judged now. A genuinely read-only overlay
has no upper and is left alone. Reading the upper layer needs overlayfs'
private struct ovl_fs/ovl_layer, which are absent from vmlinux.h when
overlayfs is built as a module; they are declared as CO-RE type flavors and
guarded by bpf_core_type_exists(), and the service now orders itself after
modprobe@overlay.service so the module's BTF is present by the time the
program is loaded.
A managed (nsresourced-provisioned) user namespace can therefore only mount
a writable overlay whose upper dies with it (e.g. a tmpfs it set up
itself). Callers that need a writable, persistent (non-tmpfs) upperdir --
mkosi, for one -- should use a regular unprivileged user namespace for that
rather than a transient nsresourced range, since such an overlay is refused
for a managed namespace by design.
test-userns-restrict now mounts real overlays: a read-only overlay, a
writable overlay with a tmpfs upper, and a writable overlay whose upper is
an init-owned but idmapped mount are all allowed, while a writable overlay
with a plain init-owned upper is refused with EPERM. The kernel permits that
last mount without the LSM loaded, so the EPERM is the policy's doing.
vmspawn's positional arguments are extra kernel command line arguments
rather than a command to execute, so unlike nspawn or run it has nothing
to shield from the option parser: a kernel command line argument never
begins with a dash. Stopping at the first one only means an option that
follows one is silently taken for another kernel command line argument,
with no diagnostic:
systemd-vmspawn -i image.raw console=ttyS0 --set-credential=foo:bar
boots with "--set-credential=foo:bar" appended to the kernel command
line instead of setting a credential.
The stop was right when it was written. In 9de3cc1484, vmspawn's first
commit, the trailing arguments were appended to the QEMU command line
directly, so they really were options that had to reach QEMU unparsed.
4291f4461e turned them into extra kernel command line arguments passed
through SMBIOS three weeks later, and updated the man page to say so,
but left the "+" in the optstring behind; 4c778c51c0 and 5ff0ccaf8b then
carried it into the new option parser as
OPTION_PARSER_STOP_AT_FIRST_NONOPTION.
Use the default parser mode, which permutes options ahead of the
arguments; "--" still ends option parsing for an argument that does look
like an option. test-options gains the case this turns on, an option
after a positional argument in the default mode, beside the existing one
for the mode vmspawn no longer uses.
tar_c() keeps a database of already seen inodes so it can emit hardlinks
rather than storing the same content repeatedly. It prefers to keep that
database on a tmpfs superblock of its own, which is created lazily on the
first hardlinkable inode and which requires CAP_SYS_ADMIN in the caller's
user namespace. Failing that it falls back to a temporary directory under
/var/tmp/.
In the tar child forked off by import_fork_tar_c() the tmpfs branch can
never succeed: the child joins the user namespace, then drops all
capabilities but CAP_DAC_OVERRIDE, and only afterwards runs tar_c(), by
which point CAP_SYS_ADMIN is long gone. Every export therefore silently
took the /var/tmp/ fallback.
That fallback is a problem beyond being the slower path. The child runs
with the transient UID range assigned to the user namespace, so the
directory it creates is owned by a recyclable UID on a file system that
outlives the namespace, which is exactly what the nsresourced BPF-LSM
policy exists to prevent. With that policy in effect the mkdtemp() is
refused and the export fails with EPERM.
Create the database up front, in the window after the user namespace has
been joined and before the capabilities are dropped, and pass the
resulting fd into tar_c(). The database then lives on a superblock that
dies together with the namespace, no persistent file system is touched,
and the sandbox around tar is unchanged.
tar_c() keeps its lazy fallback for the callers that do not pass an fd,
i.e. systemd-dissect and the test, which do not run under a transient
range. The fd is borrowed, not consumed.
Signed-off-by: Daan De Meyer <daan@amutable.com>
Combining --ephemeral with --grow-image= truncated the file passed to
--image= to the requested size before the VM was started, so a mode whose
entire purpose is to leave the original image untouched modified it on
disk.
Skip growing the original image when running ephemeral, and instead create
the qcow2 overlay with the requested virtual size, so the guest sees the
larger disk while the base image stays read-only. As only the overlay is
sized in that case, the base image format no longer matters, so allow
--grow-image= together with qcow2 images when running ephemeral. The round
up to a multiple of 4096 moves to option parsing so both paths apply the
same value.
The vmspawn drive test grows an ephemeral VM and checks that the image
passed to --image= keeps its size, and that the overlay QEMU runs off is
created at the requested one.
Reflinking is a safe optimization whenever the source and destination are
regular, seekable files on a filesystem that supports cloning. Requiring each
caller to opt in through COPY_REFLINK adds flag plumbing without changing the
necessary fallback behavior.
Drop COPY_REFLINK, renumber the remaining flags, and have copy_bytes_full()
unconditionally try FICLONE or FICLONERANGE before entering its normal copy
loop. Isolate the clone attempt and its file-offset bookkeeping in a helper so
copy_bytes_full() only has to handle its cloned, unavailable, and error results.
A successful clone therefore completes in one operation, while an unsupported
or rejected clone falls back to progress-limited copy_file_range(), sendfile(),
or buffered copying. Keep the public reflink helpers for operations that
specifically require clone semantics.
Existing test-copy coverage exercises both bounded and unbounded regular-file
copies through copy_bytes_full().
Signed-off-by: Daan De Meyer <daan@amutable.com>
systemd-repart currently forces newly created image files into NOCOW mode.
That prevents files from being reflinked into the image, making image builds
slower and increasing their disk usage on filesystems that support cloning.
Add a tristate --cow= option. By default, leave the filesystem or parent
directory COW policy unchanged. With --cow=yes, explicitly enable COW; with
--cow=no, retain the previous behavior of forcing NOCOW. Add XO_COW as the
counterpart to XO_NOCOW so xopenat_full() applies either policy while retaining
its normal creation-error cleanup.
Document the new option and extend TEST-58-REPART to verify inherited COW and
NOCOW policies as well as explicit COW and NOCOW overrides. Compare the unset
behavior with the filesystem default so the test also works on nodatacow
mounts, and skip it when the inode attribute is unsupported.
Signed-off-by: Daan De Meyer <daan@amutable.com>
File names containing backslashes cannot be checked out on Windows,
are not handled properly by build systems such as buck, and are awkward
to work with in general, so store the \x2d slice units under sanitized
file names, and rename them to the real unit name on install via a new
optional "name" key in the units list.
The fuzz-unit-file corpus sample is renamed as well; its file name is
not meaningful to the fuzzer, and dropping the backslash means it is no
longer skipped by the meson workaround for
https://github.com/mesonbuild/meson/issues/1564, so it runs as a
regression test again.
Commit 8355eb6e11 ("meson: Check if files returned by git ls-files
actually exist") checks the paths printed by git ls-files, which are
relative to the project root, with fs.exists(), which resolves them
relative to the test/fuzz subdirectory. As a result, every corpus
sample was silently skipped, and only the generated directives tests
were registered. Resolve the paths against the project source root
instead.
The zstd blob decompression code assumed ZSTD_getFrameContentSize() always
returns the decompressed size, which holds for the journal and coredump blobs
systemd compresses itself with the one-shot ZSTD_compress(). Frames produced by
the streaming API, however, don't record the decompressed size in their header,
so ZSTD_getFrameContentSize() returns ZSTD_CONTENTSIZE_UNKNOWN. Per the zstd
documentation this is not an error, but decompress_blob_zstd() and
decompress_startswith_zstd() bailed out with -EBADMSG regardless.
This broke kexec of kernel images compressed with 'zstd' (e.g. zstd -22), whose
ZBOOT payload is decompressed via decompress_blob().
Treat only ZSTD_CONTENTSIZE_ERROR as fatal and, when the size is unknown, grow
the output buffer as we stream the frame out instead of relying on the recorded
size. Add a regression test that feeds a streaming-compressed (hence
unknown-size) zstd frame through decompress_blob() and decompress_startswith().
Let's translate openssl's errors to proper errnos
where we can instead of returning EIO for everything.
Let's also make log_openssl_errors() public so we can
use it everywhere and migrate the rest of the codebase
to use it.
The previous policy was primarily written from a standpoint
that AI models are not very good and we didn't wanna waste any
time reviewing PRs generated by AI. Now that AI models have become
actually good and their output is just as good as regular contributions,
let's stop requiring the disclosure as its pointless to still have it,
it doesn't really matter anymore whether a patch was written with or without
AI. It's up to the author to make sure they're not wasting our time by
submitting unreviewed, untested code upstream, regardless of whether that
code is written by an AI or not.
The new policy is inspired by https://github.com/lxc/incus/pull/3506, with
various removals to be less adverse to the usage of AI.
The only runtime trigger for registry cleanup is the BPF kprobe that fires
on user namespace destruction; when it is missed (ring buffer overflow,
kprobe missing, fdstore entry dropped), the dead namespace's registry entry
survives and keeps its UID/GID ranges blocked until the manager restarts and
its startup sweep runs. The allocation hot path checked whether a candidate
range was already taken but never whether the namespace holding it was still
alive, so a single dead namespace could permanently starve an allocation.
This is most visible when a parent delegates its entire container UID window
to a child that then dies: every subsequent allocation from the parent fails
with NoDynamicRange even though the ranges are reclaimable.
Add userns_registry_reap_if_dead(), which probes a registered namespace's
liveness via the kernel namespace identifier recorded at allocation time and,
if it is authoritatively dead, releases its registry entry — restoring any
ranges it received via delegation to their ancestors. Call it from the
allocation availability check for both transient registrations and delegated
ranges, walking a chain of dead ancestors in the delegation case. This
mirrors the existing inode-slot stale cleanup and makes allocation
self-healing without waiting for a restart.
The startup sweep grew the same load-probe-release logic, so route it through
the new helper too; its errno return distinguishes alive, no-recorded-id, and
unprobeable-environment cases so the sweep keeps its early-out when lookup by
id isn't possible at all.
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
663f0bf5cb stopped reusing the original block device fd whenever
partition scanning was requested (LO_FLAGS_PARTSCAN) but couldn't be
enabled on the device, so that nested partition tables on devices the
kernel won't scan (e.g. the pmOS/android case) get exposed via a real
loop device.
However that also forced a pointless loop device for any partition that
carries a file system directly, e.g. a btrfs subvolume mounted via
MountImages=. For multi-device btrfs this is fatal: the kernel rejects
seeing the same member via both the original partition and the loop
device, and the mount fails.
A loop device is only ever needed here to expose a nested partition
table. So only refuse the shortcut when the device actually carries one,
probed via gpt_probe(), instead of whenever partition scanning is
disabled. Devices carrying a file system directly (or nothing) take the
shortcut as before.
Add an integration test to cover the failure scenario of the original
issue.
Fixes: https://github.com/systemd/systemd/issues/42520
Replaces: https://github.com/systemd/systemd/pull/42576
Follow-up for 663f0bf5cb79ecaf6dd71441ecdc9dc401e7eae6
Co-Authored-By: Luca Boccassi <luca.boccassi@gmail.com>
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
probe_gpt_boot_disk_needs_loop() sets ID_PART_GPT_AUTO_ROOT_DISK_NEEDS_LOOP
for any whole disk that holds the boot ESP/XBOOTLDR but whose partition table
the kernel cannot parse. Until now the udev rule turned that into a
systemd-loop@.service for every block device.
That is too broad: device-mapper devices also report kernel partition
scanning as disabled, but their partitions are managed in userspace by kpartx
(see 66-kpartx.rules). Setting up a loop device on top of them re-exposes the
same partition table a second time and only causes trouble.
Restrict the rule to optical drives, the one class that genuinely needs a
kernel-side loop device (El Torito GPT sector size mismatch, or drives that do
not support partition scanning) and that has no userspace partition manager of
its own.
Co-developed-by: Claude Fable 5 <noreply@anthropic.com>
Since 4e0eabd401 ("udev: also trigger loop device for boot disk when
partition scanning is unsupported"), builtin_blkid() bails out entirely as
soon as probe_gpt_boot_disk_needs_loop() reports that a loop device is
needed, skipping all superblock probing. As a result whole-disk properties
such as ID_PART_TABLE_UUID and ID_FS_* are no longer set.
This regresses any whole disk whose partitions the kernel cannot expose
itself but which is otherwise perfectly probeable, most notably
device-mapper multipath disks: kernel partition scanning is disabled on them
(their partitions are managed in userspace by kpartx), so they are now
flagged as needing a loop device and lose their ID_PART_TABLE_UUID.
The early return was never necessary. The original intent was only to skip
root partition discovery on the device, and that already happens on the loop
device instead: find_gpt_root() bails when the kernel can't scan partitions,
blkid probes at the device's own logical sector size so a GPT written for a
different sector size is simply not detected, and PART_ENTRY_* is only
emitted for partitions the kernel actually registered, of which a
loop-needing whole disk has none. So keep probing the device for its
whole-disk properties unconditionally and let partition and root discovery
happen on the loop device.
Co-developed-by: Claude Fable 5 <noreply@anthropic.com>
Switch the model used by the Claude review workflow from Opus 4.8 to
Fable 5, both for the top-level invocation and the review subagents.
Co-developed-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A bind mount inherits the mount flags of the file system the source
directory resides on. For root=bind: the source typically lives below
/run/ (e.g. a freshly unpacked tar image in /run/machines/), which is
mounted nosuid,nodev, so those flags propagated to /sysroot and broke
suid binaries (e.g. sudo) and device nodes on the booted system.
Default bind root mounts to dev,suid,exec instead, unless the user
overrides this via rootflags=.
Fixes: https://github.com/systemd/systemd/issues/41352
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
keyfile-erase only erases a key file explicitly configured in the third
field of crypttab. Key files automatically discovered in
/etc/cryptsetup-keys.d/ and /run/cryptsetup-keys.d/ are considered shared
resources not owned by an individual volume, and are never erased. Make
this explicit in the documentation and add a code comment clarifying the
intent.
Fixes: https://github.com/systemd/systemd/issues/41127
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
The orchestrator repeatedly emitted StructuredOutput with only a long
`summary` and no `comments`, which the schema rejects as missing a required
property; one run burned 12 retries (and a large share of its output tokens)
re-typing rejected summaries before it shrank the summary enough to include
`comments`. Instruct it to build `comments` first, always include `comments`
and `resolve` (even when empty) in a single call, and keep the summary concise
so the detailed prose lives in the comments rather than being duplicated.
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
Change the review fan-out from one subagent per commit to one subagent per
lens, each reviewing every commit through a single perspective. Four base
lenses (correctness/memory safety, lifetimes/concurrency, security, API/style)
always run; the orchestrator skims the diff and adds 1-3 PR-specific lenses
(e.g. a DNS protocol lens for resolved changes). A single generalist reviewer
tended to converge on one finding on large diffs; focused lenses dig deeper.
Commits are reviewed in chronological order via a commit-order.txt manifest,
since the SHA-named worktree dirs don't sort chronologically.
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
Bump the Bedrock model ID to us.anthropic.claude-opus-4-8 (the -v1 suffix
was dropped after 4.6), pin ANTHROPIC_DEFAULT_OPUS_MODEL so the review
subagents resolve to 4.8 as well, and switch the effort level from max to
xhigh.
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
When ssh-ing into a VM, you generally do not want to log
in as your user from the host. Let's default to the root
user unless a user is explicitly specified.
sshd reads AuthorizedKeysFile after dropping to the authenticating user's UID, so the
0400 credential file under %d/ is unreadable for non-root users. Materialize a 0444
copy in a RuntimeDirectory so the ephemeral key works for any user. */
Not the case on the sparc architecture. Replace with two translation
functions. The poll to epoll variant isn't used yet but will be used
when we add io-uring support to sd-event so we already add it here so
that it is in place.
Follow-up for cf4c65afa8
This test can regress without changes on our end when
an older systemd is built against a newer glibc and
the newer glibc has an ABI break for a symbol we use
but don't dlsym(). To avoid spurious test failures for
downstream packagers, let's only run the test in developer
mode for now.
On architectures where glibc support was introduced after our baseline
(e.g. loongarch64, added in glibc 2.36), every imported symbol is
tagged with a version newer than the baseline, so the baseline check
fails spuriously. Detect this by taking the minimum version observed
across all binaries: if it exceeds the baseline, treat that minimum
as the effective baseline instead.
Follow up for d9600a2a
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
EFI_GUID requires 4-byte alignment due to its uint32_t Data1 field, but
callers may pass pointers at arbitrary offsets into serialized EFI
variable buffers (e.g. bootctl walking BootXXXX entries). UBSan flagged
the misaligned member access; the old comment claiming the struct was
packed was wrong. Copy the bytes into an aligned local first.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
When a user namespace dies and its registry entry is torn down, the kernel
can recycle its inode number for a freshly-created namespace. A subsequent
registration or operation request can therefore find a stale registry entry
keyed by the same inode that actually belongs to a different, now-dead user
namespace.
Use NS_GET_ID to compare the kernel-assigned namespace identifier against
the stored one whenever we look up the registry from a live userns fd
(AddMount/AddControlGroup/AddNetworkInterface, plus the two registration
paths). Extract release_userns_by_info()/release_userns_by_inode() into
userns-registry.c so nsresourcework can fully clean up stale entries
(BPF allowlist, fdstore fd, cgroups, netifs, on-disk record) before reusing
the slot, and remove the now-unused userns_registry_inode_exists().
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Traditionally, asynchronous programming in systemd has been achieved
using
sd-event along with the asynchronous interfaces of sd-bus and
sd-varlink.
This works well when the system is reacting to events and all code
triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback
can
use the stack as usual, declaring local state and ensuring proper
cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block
execution
within the callback, we can't directly invoke a long-running operation
and
wait for its result without introducing complexities. Instead, we need
to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along
as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope
may
be prematurely cleaned up. Therefore, the struct must store copies of
the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a
loop,
the complexity grows further. We must introduce additional shared state
to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must
track
the lifecycle of each long-running operation in the global Manager
struct,
ensuring proper cleanup even when stack unwinding can no longer manage
the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s
running
synchronously, but with the ability to yield control at predefined
points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while
waiting
for the long-running operation to finish and then "resume" the operation
once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in
a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve
the
handling of complex workflows, making the code easier to write and
maintain.
The implementation is based on ucontext.h's makecontext() (with a
fallback
to the venerable sigaltstack() approach on musl),
sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can
switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext()
because the
latter forcibly saves/restores a per context signal mask every time it
is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and
maintain
a per thread signal mask, which makes much more sense than having a per
fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used
until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a
deferred
event source which resumes the fiber when enabled. The deferred event
source
is oneshot by default so the fiber will run immediately until it yields
or
suspends. If it yields, the deferred event source is enabled again
(oneshot)
immediately. If it suspends, before it suspends, one or more event
sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for
completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or
any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is
cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the
fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup
to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them
behind
a generic futures interface to represent long-running operations,
regardless of
whether those operations are running on a fiber or not. Aside from
fibers, the
futures library (sd-future) will for example allow waiting for sd-event
sources
and doing sd-bus calls in the background as well. Fibers can suspend
until a
future is ready with sd_fiber_await() or by having the future wake up
the fiber
explicitly in its callback. A future always defaults to waking up the
current
fiber.
Each future kind plugs into the library by providing an sd_future_ops
vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it
via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to
that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from
a
fiber via sd_fiber_await() (synchronous-looking fiber code).
sd_fiber_await()
is itself built on a "wait future" that resolves when its target
resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber
callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future
impl's
cancel callback, which is responsible for tearing down its work and
ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from
sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new()
—
take a self-reference on their future so they outlive the caller's
scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds
would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal
sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an
unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the
following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of
it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
On 32-bit userspace running against a 64-bit kernel
BTRFS_IOC_GET_SUBVOL_INFO returns -ENOTTY: struct
btrfs_ioctl_get_subvol_info_args embeds four btrfs_ioctl_timespec
values, and that timespec struct (__u64 sec; __u32 nsec) packs to 12
bytes on i386 but 16 on x86_64 due to differing __u64 alignment.
sizeof(struct) is part of the ioctl cmd number via _IOR(), so the cmd
emitted by 32-bit userspace doesn't match the case label compiled by
the 64-bit kernel and the switch falls through to -ENOTTY.
btrfs already handles this exact class of bug for
BTRFS_IOC_SET_RECEIVED_SUBVOL via a btrfs_ioctl_timespec_32 struct
plus a _32 cmd alias in fs/btrfs/ioctl.c, but GET_SUBVOL_INFO (added
in 2018, four years after that fix) didn't get the same treatment.
Until a kernel patch lands the test can't exercise the ioctl on
32-bit, so convert TEST(info) to TEST_RET(info) and return
EXIT_TEST_SKIP with a clear message when -ENOTTY comes back. The
other tests in the file use ioctls that already have working compat
paths and remain unaffected.
core: add User,Group,SupplementaryGroups,Nice to varlink
Unit.StartTransient
This commit adds more writable fields to the
io.systemd.Unit.StartTransient
varlink method. With this its possible to set:
User,Group,SupplementaryGroups,Nice values.
Plus tests for them.
---
core: report unsupported service fields in varlink calls
Just like for the unsupported/bad exec_fields we should show
a message about what field is bad for service parameters. This
commit adds it using the same pattern. The JSON parser works in
fail-fast mode so we only display the first bad field (and
it depends on the parser what it finds first).
Add -fexcess-precision=standard so gcc inserts ISO C99 conformant
rounding at assignments, casts, and returns — without it, double values
on x87 happily stay at 80-bit extended precision across operations and
diverge from the SSE/x86_64 behavior, making strict equality comparisons
architecture-dependent.
The flag doesn't fully cover x87: per gcc PR#323
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=323), a function return
value carried in ST(0) can arrive at the caller still at 80-bit, so a
double that ought to compare equal to a same-magnitude literal picks up
extra mantissa bits and doesn't. Wrap fp_equal in volatile-double
temporaries to force a memory roundtrip — the only operation that
reliably truncates on x87 — so its callers get consistent results
regardless of how the operands were produced.
Add a TEST(fp_equal) case that exercises the previously-broken pattern:
a runtime 1.0/10.0 computed inside a noinline helper, returned across
the function ABI boundary, then compared against the literal 0.1.
Without the volatile truncation this assertion fails on 32-bit gcc.
The mock servers used to be driven out-of-band: each test created a
socketpair, forked a child, ran a hand-coded request/response script
against the raw fd, and sent SIGTERM to tear it down. That worked but
required pidref/process-util/signal plumbing in every test, two
distinct execution contexts that couldn't share state, and a JsonStream
attached to the mock side that pretended to be event-loop-driven while
actually being driven manually via blocking reads.
Now that JsonStream suspends when on a fiber, the mocks can live
inside the same process and event loop as the client. Each mock is
rewritten as an sd-fiber that runs alongside the client fiber: so the
mock fiber yields on I/O and the event loop schedules the client in the
meantime. Both sides progress cooperatively, no fork/SIGTERM/PID tracking,
no manual phase tracking.
Two cleanups fall out of the rewrite:
- A QMP_TEST(name, mock_fn) { ... } macro encapsulates the per-test
scaffolding (event loop, socketpair, mock fiber spawn, exit-on-idle
shim) and injects an already-connected QmpClient *client into the
test body. Each test now reads as a flat sequence of
qmp_client_call() invocations against that client.
- Repeated mock command/reply scripting is factored into
mock_qmp_expect(), mock_qmp_reply(), mock_qmp_expect_and_reply(),
mock_qmp_handshake(), and mock_qmp_query_status_running(). The
greeting JSON is built with sd_json_buildo() instead of being parsed
from a literal.
The file shrinks from 756 to 494 lines, mostly through deletions.
The synchronous qmp_client_call() pumps the event loop until its reply
arrives, pinning the parsed reply on c->current so it can hand out
borrowed pointers to the caller. That model only fits one in-flight
sync call: a second qmp_client_call() on the same client clears
c->current before issuing its own send, invalidating the first
caller's borrowed pointers. On a single-threaded event loop that was
fine, but with fibers two concurrent calls on the same client can
interleave through the pump (json_stream_wait() suspends the running
fiber) and trample each other.
To fix this, make qmp_client_call() detect when it's running on a fiber
whose event loop matches the client and transparently delegate to
qmp_client_call_suspend(), which makes use of a new QmpFuture to allow
multiple concurrent calls to qmp_client_call().
To make this work concurrently, we also change qmp_client_call() to
hand out references and copies of errors so that we don't have to store
the borrowed pointers we hand out in the QmpClient struct.
Add varlink_server_bind_fiber() and varlink_server_bind_fiber_many()
in varlink-util.{c,h} for registering a method handler that should
run on a dedicated fiber per dispatch. The fiber-bound methods live
in a separate s->fiber_methods map alongside the regular s->methods;
bind_internal()/bind_many_internal() are factored out so the regular
and fiber bind variants share their parsing/insertion code.
Registering the same method in both maps is rejected because the
dispatcher consults the regular map first and would otherwise
silently shadow the fiber binding.
varlink_dispatch_fiber() builds a VarlinkFiberData (refs to the
connection, parameters, and method name), spawns a fiber via
sd_fiber_new(), and makes the future floating so the fiber
self-manages its lifetime — neither the dispatcher nor the
connection has to track it. The fiber's priority is set to one
below the connection's quit event source so that on graceful
shutdown the fiber's exit handler fires (and runs its cleanup)
before varlink's quit_callback() closes the connection underneath
it; this is what lets a fiber-bound handler reply or flush its
sentinel on a still-open connection during shutdown.
The connection state transitions are reordered so they happen before
the fiber spawn rather than after the synchronous callback returns:
the fiber runs after dispatch has already moved past PROCESSING, which
matches the behaviour expected for a deferred reply (the fiber may
either reply immediately, or stash the connection and reply later, in
which case the post-callback logic treats it as a PENDING_METHOD).
Note that all the synchronous varlink APIs (sd_varlink_call() and friends)
already behave properly when on a fiber because they call json_stream_wait()
which calls ppoll_usec() which we already fixed to suspend when called from
a fiber.
The client/server varlink tests are migrated to fibers (threads → mock
server fibers on the same event loop) to exercise the new paths.
Two changes to teach sd-bus how to behave when called from a fiber, in
order of increasing depth:
2. sd_bus_call() now redirects to a new bus_call_suspend() helper when
the caller is a fiber whose event loop is the same one the bus is
attached to. The plain bus_poll() path serializes all bus traffic on
the slot's reply (only one method call can be in flight per
sd_bus*), which would defeat the point of running multiple fibers
against one bus. bus_call_suspend() builds on the async sd-bus API:
it wraps the call in a new BusFuture (sd-bus/bus-future.{c,h}) that
resolves when the reply or method-error arrives, lets the fiber
await that future, and surfaces the reply to the caller via
future_get_bus_reply(). Because the futures live on the event loop
rather than a per-bus slot, multiple fibers can drive concurrent
method calls against the same bus.
3. A new private SD_BUS_VTABLE_METHOD_FIBER flag dispatches a vtable
method handler on its own fiber, so handlers are free to use
sd_bus_call() against the same bus, sd_fiber_sleep(), loop_read(),
etc. without stalling the event loop for other connections or
handlers. The flag stays out of sd-bus-vtable.h (its bit value is
reserved there to prevent collisions) — the fiber runtime is a
systemd-internal implementation detail.
Lifecycle of fiber-dispatched handlers is tracked on the bus itself: a
new bus->fiber_futures set holds a ref to each in-flight handler.
bus_enter_closing() cancels every entry and process_closing() returns
with the bus still in CLOSING state until the set drains, so we can be
sure no fiber handler outlives the bus. bus_fiber_resolved() removes
the entry on completion. bus_free()'s assert(set_isempty()) makes the
invariant load-bearing.
Note that plain sd_bus_call() already works correctly on a fiber as it
calls ppoll_usec() which has already been modified to suspend when
running on a fiber.
To exercise these changes the existing thread-based client/server
sd-bus tests (test-bus-chat, test-bus-objects, test-bus-peersockaddr,
test-bus-server, test-bus-watch-bind) are migrated to fibers, and a
new test-bus-fiber is added that covers SD_BUS_VTABLE_METHOD_FIBER —
including handlers that issue nested sd_bus_call() on the same bus, the
cancel-on-close path, and concurrent dispatches across multiple fibers.
sd_event_run() blocks the calling thread on the event loop's epoll fd
until something happens. When the caller is a fiber, that's the wrong
behaviour: blocking the thread also stalls every other fiber and the
outer event loop driving them. The most common way to hit this is a
fiber that creates its own inner event loop (e.g. a server-style fiber
that wants to dispatch its own sources independently of whatever loop
the test or supervising fiber is running on) — with the existing
implementation the inner sd_event_run() would hold the thread while the
outer scheduler should be free to advance other fibers.
Add an event_run_suspend() variant in sd-event/event-future.c that
performs the same prepare/wait/dispatch dance, but when the fast path
finds nothing ready it (a) creates an IO future watching the inner
event loop's epoll fd on the *outer* event loop, (b) optionally creates
a time future for the timeout, and (c) suspends the fiber. When either
future fires the fiber is resumed and the prepare/wait/dispatch sequence
runs once more to actually dispatch what's pending. sd_event_run()
checks sd_fiber_is_running() and delegates to this variant when on a
fiber; profile_delays accounting is intentionally skipped on that path
since the underlying prepare/wait/dispatch primitives already account
for themselves.
Some helpers in src/basic — ppoll_usec_full() (used by fd_wait_for_event()),
loop_read(), loop_read_exact(), loop_write_full() and
pidref_wait_for_terminate_full() — block the calling thread. That's the
right behaviour outside a fiber but not inside one, where blocking the
thread also stalls every other fiber running on the same event loop.
Rewriting every caller to pick a fiber or non-fiber variant explicitly
would be a lot of churn and would split otherwise-shared code paths in
two.
Instead, the helpers detect at runtime whether they're running on a fiber
and dispatch to a suspending variant when they are. FiberOps in
fiber-ops.h holds five function pointers (ppoll, read, write, timeout,
cancel_wait_unref); a fiber_ops global constant is populated whenever we
enter a fiber with functions that delegate to suspending variants of common
syscalls. With this approach, the variants themselves stay in libsystemd
which is required because they make use of sd-event.
- loop_read()/loop_read_exact() take the fiber read hook on a fiber
unless the caller asked for a non-blocking attempt (do_poll=false) and
the fd is already non-blocking — in that case we fall through to read()
to preserve the existing return-EAGAIN-immediately semantic. The hook
itself suspends on EAGAIN until data is available, so neither the
do_poll knob nor the explicit fd_wait_for_event() retry loop are
needed on the fiber path.
- loop_write_full() likewise takes the fiber write hook on a fiber,
except when timeout=0 with an already-non-blocking fd (preserving the
fast-return-EAGAIN semantic). The fiber path runs inside a
FIBER_OPS_TIMEOUT() scope so the caller's timeout is honoured via a
deadline future, mirroring SD_FIBER_TIMEOUT() but reachable from
src/basic without pulling in sd-future.h.
- pidref_wait_for_terminate_full() polls the pidfd via fd_wait_for_event()
before each waitid() when either a finite timeout is set or we're on a
fiber, and requires pidref->fd >= 0 in those cases (returning
-ENOMEDIUM otherwise — extending the rule that already applied to
finite timeouts). The poll suspends the fiber via the ppoll hook above;
the subsequent waitid() doesn't block because the pidfd is already
signalled.
Add a family of sd_fiber_*() I/O wrappers that, when called from a
fiber, behave like blocking I/O from the caller's perspective but
yield to the event loop instead of blocking the thread:
sd_fiber_read / sd_fiber_write
sd_fiber_readv / sd_fiber_writev
sd_fiber_recv / sd_fiber_send
sd_fiber_connect
sd_fiber_recvmsg / sd_fiber_sendmsg
sd_fiber_recvfrom / sd_fiber_sendto
sd_fiber_accept
sd_fiber_ppoll
Most of them share a single helper, fiber_io_operation(), which when
invoked outside a fiber falls through to the underlying syscall
directly, preserving the regular blocking behaviour. Inside a fiber
the helper flips the fd to non-blocking (restoring its original mode
on return), tries the syscall once on the fast path, and on EAGAIN/
EWOULDBLOCK creates an sd-event-backed IO future via future_new_io(),
suspends the fiber, and retries the syscall once the event source
fires.
future_new_io() itself is added to sd-event/event-future.{c,h} as a
new IoFuture kind. It wraps sd_event_add_io() into an sd_future:
oneshot enable, EPOLLERR translated via SO_ERROR (suppressed for
non-sockets), and the fd duplicated with F_DUPFD_CLOEXEC to avoid
EEXIST when multiple sources watch the same descriptor.
Together these let fiber-using code write straight-line socket and
pipe I/O without bundling state into callbacks.
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
Drop the -fundamental suffix from src/fundamental/ headers in favor of names
that match their src/basic/ or src/shared/ counterparts (e.g.
macro-fundamental.h -> macro.h, assert-fundamental.h -> assert-util.h,
cleanup-fundamental.h -> cleanup-util.h). Rename src/basic/{btrfs,label}.{c,h}
to use the -util suffix to match the existing shared/btrfs-util and
shared/label-util siblings. Rename src/shared/mkdir-label.{c,h} to mkdir.{c,h}
and src/shared/tmpfile-util-label.{c,h} to tmpfile-util.{c,h} to match the
corresponding src/basic names.
This saves us from having to come up with separate names for files that do
the same thing across tiers, and it makes it easier to move stuff between
src/fundamental, src/basic and src/shared: consumers just #include "foo.h"
and pick up whichever tier their -I path resolves to first, so call sites
don't need to be updated when an API moves between layers.
Where a higher-tier wrapper exists (e.g. src/basic/macro.h wrapping
src/fundamental/macro.h), the wrapper uses an explicit "../fundamental/foo.h"
or "../basic/foo.h" relative include for the lower-tier header. We can't use
GCC's #include_next directive for this — when the wrapper is reachable both
via same-dir-as-source lookup and via -I (e.g. -Isrc/shared) for the
directory it lives in, #include_next advances by exactly one slot in libcpp's
internal directory chain and lands on the same physical directory it was
already in, never reaching the lower-tier sibling (see make_cpp_dir() in
gcc/libcpp/files.cc:1986).
To make sure the right headers are always picked up, the include directories
are reordered so that e.g. src/shared always takes priority over src/basic and
similar for the other directories.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Drop the standalone Unit-tests (musl) workflow that ran on an Alpine sandbox
spun up by jirutka/setup-alpine, and merge it into unit-tests.yml as a new
build-musl job that provisions a postmarketOS tools tree via mkosi and runs
the meson build + test suite through 'mkosi box'. postmarketOS is musl-native,
so the musl-gcc / -idirafter /usr/include wrappers the Fedora tools tree
needed are gone; the linter.yml's own musl build step also goes away since
the unit-tests workflow now covers it (and tests it).
postmarketOS doesn't ship a downstream systemd packaging spec, so the new
tools tree config in mkosi.tools.conf/mkosi.conf.d/postmarketos.conf does not
set PrepareScripts and lists build deps manually. mkosi.sync now early-exits
when PKG_SUBDIR is unset so the missing pkgenv entry doesn't trip set -u.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Instead of having CI runner specific checks, let's just
skip the test if we get EXIT_CGROUP which is what we get
when we can't create a cgroup. This makes the check work
independently of CI runner, and specifically also on github
actions.
if stdin is connected to a closed pipe or similar, imply
PTY_FORWARD_READ_ONLY so we don't even try to read from it
in the first place. Otherwise we'll immediately get a hangup
which will cause the forwarder to call sd_event_exit() and
shut down the event loop.
Debugged-by: Christian Brauner <brauner@kernel.org>
- test-random-util is reworked to not use sqrt()
- pretty-print.c inlines ceil() so libm doesn't have
to be linked into libshared
- We add fno-math-errno to allow inlining of more math
functions by not requiring standard math functions to
set errno on invalid input.
Run acquire_root_devno() on the varlink-provided node so symlinks (e.g.
/dev/disk/by-id/...) resolve to their canonical /dev/ path before being
used. Without this, sym_fdisk_partname() produces a "-partN" symlink
that udev hasn't created yet when repart calls open() on it right after
BLKPG_ADD_PARTITION, failing with ENOENT.
This also brings the varlink path in line with the CLI path's
partition-to-whole-disk and dm-crypt-to-backing resolution.
For every built executable, internal shared library, and plugin module,
verify two link-time properties via readelf:
1. No imported GLIBC symbol's version is newer than 2.34.
2. The dynamic section's NEEDED entries reference only glibc, the
runtime linker, our own libraries.
exp10() has a symbol version > 2.34 on latest glibc. To allow
dropping our baseline required glibc runtime version to <= 2.34,
let's add our own version to prevent pulling in the newer symbol
from glibc.
meson test --wrapper hook to print a gdb backtrace inline in the test
log when a test exits with an actual crash signal (SIGSEGV, SIGABRT,
SIGBUS, SIGFPE, SIGILL). Wired into the default add_test_setup() so it
runs automatically on every `meson test`.
Environmental terminations (SIGTERM/SIGKILL/SIGPIPE/SIGALRM) are passed
through without replay, and the original signal is re-raised so the
parent's wait() observes WIFSIGNALED rather than a plain exit code.
Each dlopen_*() wrapper kept its dl handle as a file-scope
'static void *xxx_dl = NULL;' even though only the wrapper itself
ever referenced it. Move each one inside the corresponding function
so its scope matches its actual use, leaving the rest of each
translation unit free of the unused file-scope name.
In pcre2-util.c the assert(pcre2_dl) in pattern_matches_and_log()
becomes assert(sym_pcre2_match), which carries the same invariant
(pattern_compile_and_log() ran dlopen_pcre2()).
* 77fce77807 apk: Implement repository_key_fetch for the postmarketOS distribution
* 7068ed49ab postmarketos: Add ruff to tools tree
* dea4b6bfc8 Add newline when writing machine id into /etc/machine-id
* 944b775d40 tools: add libtss2-tcti-device0 to opensuse tools tree
* d856d65d3b mkosi-initrd: Also add cryptsetup-libs explicitly to the initrd
* 1cc967c5b3 mkosi-initrd: Trim orphaned GPU/audio modules, add ACPI platform attrs
* a3e95a7c29 mkosi-tools: Add fish to misc profile
* 76b02d1f84 mkosi-tools: Add jujutsu to misc profile
* 0afe4cd254 mkosi-tools: Move gh to misc profile
* 9077634bad mkosi-tools: Add cryptsetup-libs to centos/fedora/opensuse
* 82846347af box: Drop background tinting
* 3e50b97101 mkosi-tools: Add libfido2
* 78c2784827 vmspawn: Use --ephemeral rather than copy_ephemeral()
* dc801b00a3 Added second call to update kerneltype after kernel is defined
* 0c5cc04a8b vmspawn: Forward journal-remote settings to vmspawn
* 2518468c65 nspawn: Use --forward-journal instead of running journal-remote ourselves
* d2b798d00c apk: skip removal of packages that aren't installed
- **bpf-util: rename from bpf-dlopen, unify version-specific symbol
handling**
- **cryptsetup: dlopen libcryptsetup in tokens**
- **tree-wide: dlopen libpam in pam plugins**
- **test-bus-marshal: dlopen() glib and libdbus instead of linking
directly**
- **lock-util: Simplify timeout for lock_generic_with_timeout()**
- **color-util: simplify hsv_to_rgb, fix rgb_to_hsv negative-hue wrap**
- **tree-wide: Use our own macros instead of fabs()/fmax()/fmin()**
- **locale-util: dlopen() libintl instead of linking against it**
- **home: Use log2u64() over log2()**
- **meson: drop libdl, threads, and librt dependencies**
- **libc: Use dlsym() from a constructor instead of weak symbols**
- **libc: Make sure C23 versions of strtol(), sscanf() are not used**
When _GNU_SOURCE is defined, glibc will always use c23 versions
of strtol(), sscanf() and friends if available (introduced after
glibc 2.34). Which means that any binaries built with headers
from newer glibc won't load on glibc < 2.38. To work around this,
redefine the appropriate constants to zero make sure the c99
versions are used instead.
Weak symbols still introduce a version requirement on a newer libc.
Resolve each libc symbol via dlsym(RTLD_DEFAULT) from a per-shim
constructor and cache the result in a file-scope static instead. This
avoids the version requirement, keeps the call path free of atomics
(constructors run single-threaded before main() and before any signal
handler can fire), and keeps dlsym() out of contexts where it is not
async-signal-safe.
Our baseline glibc is 2.34, which merged libdl, libpthread (the
dependency('threads') target), and librt into libc. Empty .so/.a stubs
remain for backward compatibility with old binaries, but new builds
resolve dl_*, pthread_*, mq_*, timer_*, etc. directly from libc.
On musl the same libraries are likewise empty stubs.
Drop the libdl, threads, and librt entries from every meson.build, and
remove the now-stale 'Libs.private: -lrt -pthread' from libudev.pc.in
since both flags resolve to empty link-time stubs on glibc 2.34+ and
musl.
Verified with readelf -d that libsystemd.so, libudev.so, and systemd no
longer carry DT_NEEDED entries for libdl/libpthread/librt.
dgettext() lives in libc on glibc and in libintl.so.8 on musl with
gettext. Resolve it via dlsym() so neither configuration produces a
hard link-time dependency on libintl: try libintl.so.8 first and fall
back to RTLD_DEFAULT (which finds dgettext in libc on glibc).
The _() macro now expands to a runtime check that returns the
untranslated string if dlopen_libintl() has not run successfully, so
callers don't have to gate every translatable message on a runtime
check. pam_systemd_home — currently the only consumer of _() — calls
dlopen_libintl() best-effort from each PAM entry point.
The meson find_library('intl') dance is replaced with a has_header()
check; the only thing we need at build time is the prototype.
To make this work, ABS() is made generic so it also works on
floats and doubles.
While at it, fold the __ABS_INTEGER indirection and the
assert_cc(sizeof(long long) == sizeof(intmax_t)) away. The previous
form switched between __builtin_llabs (clang) and __builtin_imaxabs
(gcc), with the assert keeping the two paths behaviorally identical
on every platform we build for. imaxabs was originally chosen because
intmax_t is conceptually the widest signed integer type the platform
exposes, but the _Generic ABS already casts to (long long) before the
call, so the extra width imaxabs could in theory carry was being
narrowed away immediately anyway. With both paths collapsed to
__builtin_llabs((long long) (a)), the size relationship between
long long and intmax_t is no longer relevant.
Also add explicit unsigned long long / unsigned long / unsigned int
cases that pass the argument through unchanged. The previous default
branch cast unsigned values to (long long); for values above LLONG_MAX
this reinterprets them as negative, and __builtin_llabs(LLONG_MIN) is
UB. Unsigned values are already non-negative, so passing them through
is both correct and avoids the narrowing. Smaller unsigned types
(unsigned char, unsigned short) still go through the default branch
but promote to int first and fit in long long losslessly.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
In hsv_to_rgb, restructure the conversion around the sector index
k = (int)(h/60) and fractional offset f = h/60 - k. The auxiliary
x value becomes c * (k & 1 ? 1.0 - f : f) and the six branches turn
into a switch on k. This drops the two xfmod() calls that were doing
the modulo work, in exchange for a single assert(h >= 0 && h < 360) —
all in-tree callers satisfy this and never relied on the wrap.
In rgb_to_hsv, the two xfmod() calls were no-ops (their arguments
were always within the divisor's magnitude). The trailing
xfmod(*ret_h, 360) appeared to be wrapping negative hues from the
r-max branch back into [0, 360), but fmod is sign-preserving so it
never did. Drop the no-ops and add an explicit +360 wrap so magenta
(1, 0, 1) now yields h ≈ 300 instead of -60.
Extend the tests to cover all six primary/secondary colors at sector
boundaries, all six sector midpoints (to catch any future inversion
of the ramp direction), the h-near-360 edge of the last sector, and
the rgb_to_hsv negative-wrap path via magenta. Switch the new and
existing integer-channel checks to ASSERT_EQ from tests.h; the
double-typed h/s/v range checks stay on ASSERT_TRUE since the
ASSERT_* comparison macros only support integer types.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Daan De Meyer
