The dditest block calls systemd-repart with Encrypt=tpm2 but without
--tpm2-public-key-pcrs=. Since systemd-stub drops
/run/systemd/tpm2-pcr-public-key.pem when booting from a signed UKI
systemd-repart auto-loads it and enrolls a signed PCR policy, and
then systemd-cryptsetup tpm2-device=auto has no matching signature file,
so unlock fails.
--tpm2-public-key= is not enough as the default kicks in then.
Follow-up for cd18656d47
mdadm --zero-superblock only wipes the MD metadata on the underlying
disks, not the LVM PV header that lives in the array data area. When
the VM is restarted and the test re-creates the array with the same
UUID, /dev/md127 exposes the old data including the LVM PV header, so
udev's 69-lvm.rules auto-triggers lvm-activate-mdlvm_vg.service which
races with the test's own pvcreate for exclusive access on /dev/md127.
Wipe the LVM signature off the MD device (and the underlying disks as
a belt-and-braces measure) to avoid the race on re-run, fixing failures
when the VM is rebooted instead of shut down.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
write_to_journal() was called via $(...) command substitution, so
SERVICE_COUNTER++ ran in a subshell and never incremented in the
parent:
[ 1492.668302] TEST-04-JOURNAL.sh[15064]: + local service=test-0-18493.service
[ 1492.725882] TEST-04-JOURNAL.sh[15064]: + local service=test-0-18009.service
[ 1492.739643] TEST-04-JOURNAL.sh[15064]: + local service=test-0-18493.service
[ 1492.774586] TEST-04-JOURNAL.sh[15064]: + local service=test-0-25540.service
[ 1492.815664] TEST-04-JOURNAL.sh[15064]: + local service=test-0-15916.service
[ 1492.867067] TEST-04-JOURNAL.sh[15064]: + local service=test-0-20327.service
[ 1492.899077] TEST-04-JOURNAL.sh[15064]: + local service=test-0-86.service
[ 1497.289715] TEST-04-JOURNAL.sh[15064]: + local service=test-0-10849.service
[ 1497.335791] TEST-04-JOURNAL.sh[15064]: + local service=test-0-18009.service
With 99999 possible unit names collisions are rare but not impossible,
so every now and then a CI run fails.
Have write_to_journal() set a global SERVICE_NAME variable instead and
call it directly so SERVICE_COUNTER actually goes up through the test.
When qemu reboots instead of shutting down after the last iteration,
the profile is already set to profile2 but the /root/encrypted.raw is
gone so the test fails. Reset the default boot entry at the end of the
test to make it robust against reruns.
Fixes https://github.com/systemd/systemd/issues/39553
The test leaves a lot of state around, and when the test is re-run,
for example due to the qemu bug that makes a VM reboot instead of
shutting down, it fails.
Do more cleanups in the traps.
[ 162.642175] TEST-70-TPM2.sh[2815]: Calculated public key name: 000b2b66edc3a466e81059286aaf38d09ea42a7a9dcdf6ba3b664c62f0cae4ce4f66
[ 162.642628] TEST-70-TPM2.sh[2815]: PolicyAuthorize calculated digest: 2caa740101f65734d50395d6abc64fa46015d40d1f5de239434578544e592a92
[ 162.643681] TEST-70-TPM2.sh[2815]: Calculated NV index name: 000b439cfa1534815bbe8d33b80c56f5a8d17d36fe94a7782b23a37b50def5fc5eaa
[ 162.645111] TEST-70-TPM2.sh[2815]: PolicyAuthorizeNV calculated digest: 69ee0e89fafe6b9df2cd6a5defbf74aa46cf6d92703e645d463549da4ba5e1a4
[ 162.645407] TEST-70-TPM2.sh[2815]: Combined signed PCR policies and pcrlock policies cannot be calculated offline, currently.
[ 162.649576] TEST-70-TPM2.sh[2815]: Releasing crypt device /dev/loop0 context.
[ 162.652433] TEST-70-TPM2.sh[2815]: Releasing device-mapper backend.
[ 162.653518] TEST-70-TPM2.sh[2815]: Closing read only fd for /dev/loop0.
[ 162.654359] TEST-70-TPM2.sh[2815]: Closing read write fd for /dev/loop0.
[ 162.654786] TEST-70-TPM2.sh[2815]: Failed to encrypt device: Operation not supported
Fixes https://github.com/systemd/systemd/issues/38241
The new "varlinkctl --more StartTransient" subtest pipes a JSON-SEQ
stream of multiple records into "jq --seq -e ...". CentOS 9
ships jq 1.6, where -e only inspects the last input record's output:
when the trailing record (the final reply) doesn't match the
"select()" filter, jq exits non-zero even though earlier records
match, so the test fails.
Use --slurp which collapses the records into an array first and
returns a single bool.
Follow-up for 1cde1cc3ba
Introduce a counter that tracks how many configuration reloads have
been successfully completed by the manager. The increment lives in
manager_reload() right after the "point of no return", so failed
reload attempts that bail out earlier (e.g. during serialization)
do not bump the counter.
It is accessible as a new ReloadCount property to
org.freedesktop.systemd1.Manager (D-Bus) and ReloadCount to
io.systemd.Manager.Describe (Varlink).
Also add an integration test for ReloadCount
that verifies that the new ReloadCount property increments by one per
daemon-reload, accumulates correctly across multiple reloads, and that
D-Bus and Varlink return identical values. Also tests that the counter
reset after a reexec.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Simon Lucido <simonlucido@meta.com>
Let's pick up all kinds of sidecars and show them, not just addons
This also fixes some issues regarding "root" directory handling.
In one context we'd resolve a directory claiming it was a "root", but it
wasn't.
Implements: https://github.com/uapi-group/specifications/pull/212
Extend io.systemd.CGroup.CpuUsage from a single per-unit nanosecond
counter to three rows distinguished by a "type" field of "total",
"user", or "system". The values come from cpu.stat's usage_usec,
user_usec and system_usec keys, read in a single keyed-attribute
fetch and cached on each CGroupInfo so each scrape only opens
cpu.stat once per cgroup.
VM-only test that exercises both shipped providers through storagectl:
verifies the well-known sockets exist, lists providers/volumes/
templates, creates and acquires volumes from each template
(sparse-file, allocated-file, directory, subvolume), attaches a loop
device to cover the block provider, and exercises the mount.storage
helper.
This commit adds a simple version of io.systemd.Unit.StartTransient
for varlink. It is similar to the dbus version, but there is a key
difference:
1. Instead of building the unit from key/value properties it
takes a structured json object "UnitContext" with a "Service" field
inside.
It is also only implementing a minimal set of what can be done with a
service.
2. No aux units (for now)
3. When called with --more the varlink socket can notify about
state changes depending on the notify{Job,Unit}Changes parameter
This aligns to the json objects/format from
https://github.com/systemd/systemd/pull/39391
and to show how the format can be shared it adds a new
(minimal) `ServiceContext` that is now part of
`io.systemd.Unit.List()`.
The sanitizer job was FUBAR on Fedora Rawhide (and RHEL 10 to some
extent) due to several changes:
- latest LLVM (v22) introduced a change that occasionally generates a
false-positive warning when running with sanitizers
- several tools had to be "ASan-wrapped" because:
- util-linux started linking against libsystemd which propagated to
other tools depending on its shared libraries (like libmount)
- libssh started depending on libfido2, which depends on libudev; this
then translated to an interesting depedency chain where tpm2 utils got a
dependency on libudev through libcurl -> libssh -> libfido2
- polkit added MemoryMax= to its service file, which is incompatible
with ASan-runs (at least with the current limits)
See the commits for more detailed descriptions.
Also, one note: the santizer job is currently still FUBAR on Fedora
Rawhide (or, more specifically, the TEST-50-DISSECT and TEST-58-REPART),
because mkfs.erofs also gained a dependency on libudev (through libcurl,
see above), but the wrapping currently doesn't work as it also depends
on libqpl which is linked with libtsan (which is incompatible with other
sanitizers). This is currently tracked in
https://bugzilla.redhat.com/show_bug.cgi?id=2461146
The verity signature partition content is a bare JSON object. Repart
pads it with zeros to fill the GPT partition. But when splitting out
the content as an individual file, the padding remains, so it's not
a valid text file.
jq started rejecting files with NUL bytes to fix a security issue:
https://github.com/jqlang/jq/commit/6374ae0bcdfe33a18eb0ae6db28493b1f34a0a5b
Trim the output when writing these files out.
In #39675 the reported fail was as follows:
5580s [ 247.559994] TEST-13-NSPAWN.sh[1858]: Exported 93%.
5580s [ 247.659002] TEST-13-NSPAWN.sh[1858]: Exported 95%.
5580s [ 247.785893] TEST-13-NSPAWN.sh[1858]: Operation completed successfully.
5580s [ 247.923727] TEST-13-NSPAWN.sh[1858]: Exiting.
5580s [ 258.300406] TEST-13-NSPAWN.sh[1074]: + machinectl import-raw /var/tmp/container-export.raw container-raw-reimport
5580s [ 258.323328] TEST-13-NSPAWN.sh[1884]: The 'machinectl import-raw' command has been replaced by 'importctl -m import-raw'. Redirecting invocation.
5580s [ 258.659982] TEST-13-NSPAWN.sh[1884]: Failed to transfer image: Remote peer disconnected
5580s [ 258.734218] TEST-13-NSPAWN.sh[1074]: + at_exit
Turns out that the real reason behind this fail is that the machine was
under heavy load due to a busy-loop from the stub init. The cause of
this is a bug in bash, where running commands that fork (i.e. not
built-ins) can cause a permanent busy-loop due to a desync in trap
handling if you send the signals to the bash process _just right_:
[ 90.855318] TEST-13-NSPAWN.sh[1074]: + machinectl poweroff long-running long-running long-running
[ 90.855318] TEST-13-NSPAWN.sh[1074]: + machinectl reboot long-running long-running long-running
[ 90.928980] systemd-nspawn[1679]: ++ touch /poweroff
[ 90.928980] systemd-nspawn[1679]: +++ touch /reboot
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + wait
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + wait
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + :
[ 90.928980] systemd-nspawn[1679]: + wait
...
$ journalctl --file TEST-13-NSPAWN-1.journal -o short-monotonic --no-hostname --grep "^\+ wait$" | wc -l
349734
So the stub-init was hammering the machine in a tight endless loop,
which then caused systemd-importd to timeout when talking to D-Bus:
[ 258.300096] TEST-13-NSPAWN.sh[1074]: + machinectl import-raw /var/tmp/container-export.raw container-raw-reimport
...
[ 258.415319] systemd-importd[1859]: Unable to request name, failing connection: Method call timed out
[ 258.483662] systemd-importd[1859]: Bus n/a: changing state RUNNING → CLOSING
[ 258.605442] systemd-importd[1859]: Bus n/a: changing state CLOSING → CLOSED
[ 258.659958] TEST-13-NSPAWN.sh[1884]: Failed to transfer image: Remote peer disconnected
Given this is not our issue, let's work around it by using just
built-ins from the trap handlers, which are not susceptible to this bug.
Resolves: #39675
`READY=1` is already correctly emitted when acquiring an update, but was
forgotten to be emitted when subsequently installing that update.
That meant that the state tracking code in `sysupdated` and hence
`updatectl` could not properly report the progress of the install
operation, and hence printed “Already up-to-date” after a successful
update installation, rather than “Done”.
Add a test to catch this in future.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Fixes: https://github.com/systemd/systemd/issues/41502
If a journal file contains a truncated ZSTD frame (i.e. a frame with
Frame_Content_Size > 0, but with not enough data in Data_Block),
ZSTD_decompressStream() would return a non-zero, non-error value. This
would then skip the error path in the ZSTD_isError() branch and we'd hit
the following assert:
$ build-local/journalctl -o cat --file zstd-truncated.journal
Assertion 'output.pos >= prefix_len + 1' failed at src/basic/compress.c:1236, function decompress_startswith_zstd(). Aborting.
Aborted (core dumped) build-local/journalctl -o cat --file zstd-truncated.journal
Let's handle this situation gracefully and return EBADMSG instead.
Also, add another journalctl invocation to the corrupted-journals test
that goes through the sd_journal_get_data() -> decompress_startswith_zstd()
code path which, among other things, covers the issue when run on the
provided journal file.
Extend fake-report-server.py with optional --cert, --key, --port
arguments for TLS support. Add a test case that generates a
self-signed certificate and tests HTTPS upload of metrics and facts.
Also exercise the --header param.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Add a fake HTTP server (fake-report-server.py) that accepts JSON POST
requests and validates the report structure, and test cases in
TEST-74-AUX-UTILS.report.sh that exercise plain HTTP upload of both
metrics and facts.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Ensure only privileged users can call the system scope machined's
APIs that get data out of a machine
Follow-up for 1bd979dddb
Follow-up for 9153b02bb5
Test the async QMP drive pipeline with real QEMU:
Test 1 (multi-drive): launches vmspawn with --image plus two
--extra-drive flags. This exercises multiple fdset allocations,
pipelined blockdev-add commands relying on FIFO ordering, io_uring
retry callbacks, and multiple device_add commands — all fired
without waiting for responses.
Test 2 (ephemeral): launches vmspawn with --image --ephemeral. This
exercises the most complex async path: blockdev-create fires a
background job, JOB_STATUS_CHANGE events are watched via the event
callback, and when the job concludes the deferred continuation fires
the overlay format node + device_add. If the continuation fails, the
root drive is never attached, the kernel panics, and vmspawn exits
without registering — so successful registration proves the pipeline
works.
Both tests use a raw ext4 image with a minimal init (sleep infinity)
and direct kernel boot. No virtiofsd needed.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Add TEST-87-AUX-UTILS-VM.vmspawn.sh that validates the QMP-varlink
bridge end-to-end using a real QEMU instance:
- Launches vmspawn with --directory and --linux for direct kernel boot
(no UEFI firmware or bootable image needed)
- Waits for machine registration with machined
- Verifies varlinkAddress is exposed in Machine.List
- Tests machinectl pause, resume, poweroff
- Exercises MachineInstance varlink interface directly via varlinkctl:
QueryStatus state verification across pause/resume, Pause, Resume
Skipped automatically if vmspawn, QEMU, or a bootable kernel is not
available. Runs as part of TEST-87-AUX-UTILS-VM in the mkosi
integration test suite.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Using the format specifier +%s%6N with GNU date is honored, and only
prints 6 digits of the nanoseconds portion of the seconds since epoch.
The uutils implementation of date does not honor this, and always prints
all 9 digits. This is a known bug[1], but can be worked around by
adapting this test to use nanoseconds instead of microseconds.
[1] https://github.com/uutils/coreutils/issues/11658
Add CpuUsage, MemoryUsage, IOReadBytes, IOReadOperations, and
TasksCurrent in a standalone socket-activated varlink service. These
metrics are gathered from the kernel via cgroup files and PID1's only
role is mapping unit names to cgroup paths — a separate process can
query PID1 once for that mapping and then read the cgroup files
directly, minimizing PID1 involvement.
The new systemd-report-cgroup-metrics service listens at
/run/systemd/report/io.systemd.CGroup and exposes:
- io.systemd.CGroup.CpuUsage
- io.systemd.CGroup.IOReadBytes
- io.systemd.CGroup.IOReadOperations
- io.systemd.CGroup.MemoryUsage (with type=current/available/peak)
- io.systemd.CGroup.TasksCurrent
This is spun out of #41078 and based on top of it. Will rebase once
that's merged.
Add CpuUsage, MemoryUsage, IOReadBytes, IOReadOperations, and
TasksCurrent in a standalone socket-activated varlink service.
The new systemd-report-cgroup service listens at
/run/systemd/report/io.systemd.CGroup and exposes:
- io.systemd.CGroup.CpuUsage
- io.systemd.CGroup.IOReadBytes
- io.systemd.CGroup.IOReadOperations
- io.systemd.CGroup.MemoryUsage (with type=current/available/peak)
- io.systemd.CGroup.TasksCurrent
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
With the new protocol upgrade support in varlinkctl client we can
now do the equivalent for the server side. This commit adds a new
`serve` verb that will serve any command that speaks stdin/stdout
via varlink and its protocol upgrade feature. This is the
"inetd for varlink".
This is useful for various reasons:
1. Allows to e.g. provide a heavily sandboxed io.myorg.xz.Decompress
varlink endpoint, c.f. xz CVE-2024-3094)
2. Allow sftp over varlink which is quite useful with the
varlink-http-bridge (that has more flexible auth mechanism than
plain sftp).
3. Makes testing the varlinkctl client protocol upgrade simpler.
4. Because we can.
Rename nspawn's --user=NAME option to --uid=NAME for selecting the
container user. The -u short option is preserved. --user=NAME and
--user NAME are still accepted but emit a deprecation warning. A
pre-parsing step stitches the space-separated --user NAME form into
--user=NAME before getopt sees it, preserving backwards compatibility
despite --user now being an optional_argument.
Repurpose --user (without argument) and --system as standalone
switches for selecting the runtime scope (user vs system service
manager).
Replace all uses of the arg_privileged boolean with
arg_runtime_scope comparisons throughout nspawn. The default scope
is auto-detected from the effective UID.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
TEST-74-AUX-UTILS has a number of subtests.
test/units/TEST-74-AUX-UTILS.capsule.sh runs first and starts and stops
capsule@foobar.service. Looking at the test, the unit is cleanly stopped.
But later test/units/TEST-74-AUX-UTILS.machine-id-setup.sh tests for
failed units. capsule@foobar.service is listed as failed, causing the
second subtest to fail.
Add the same test in test/units/TEST-74-AUX-UTILS.capsule.sh to see
if the failed test really originates from there.
When systemd-cryptenroll --tpm2-device=auto is called on a system where
a tpm2-pcr-public-key.pem exists it automatically creates tokens with a
signed PCR policy. Unlocking such a token via --unlock-tpm2-device=auto
requires a tpm2-pcr-signature.json file, which is not present.
This creates a race with systemd-tpm2-setup.service at boot: if the
service completes before the test, the key exists and the subsequent
--unlock-tpm2-device=auto calls fail, which I believe is the cause of
the test flakiness.
This also seems to mesh with the fact that this only flakes on Debian
CI, since that's built with ukify which installs a public key.
Let's hopefully fix this by passing --tpm2-public-key= to all
--tpm2-device= enrollment calls that aren't explicitly intended to test
signed PCR policy behaviour.
When systemd-cryptenroll --tpm2-device=auto is called on a system where
a tpm2-pcr-public-key.pem exists it automatically creates tokens with a
signed PCR policy. Unlocking such a token via --unlock-tpm2-device=auto
requires a tpm2-pcr-signature.json file, which is not present.
This creates a race with systemd-tpm2-setup.service at boot: if the
service completes before the test, the key exists and the subsequent
--unlock-tpm2-device=auto calls fail, which I believe is the cause of
the test flakiness.
This also seems to mesh with the fact that this only flakes on Debian
CI, since that's built with ukify which installs a public key.
Let's hopefully fix this by passing --tpm2-public-key= to all
--tpm2-device= enrollment calls that aren't explicitly intended to test
signed PCR policy behaviour.
Luca Boccassi
Simon Lucido
Lennart Poettering
Yaping Li
Michael Vogt
Frantisek Sumsal
Zbigniew Jędrzejewski-Szmek
Philip Withnall
Ivan Kruglov
Christian Brauner
Daan De Meyer
Nick Rosbrook
Christian Brauner
Chris Down