This commit exposes the last 10 high priority logs as metrics so that
the systemd-report reports them. The entries are reported as
`io.systemd.Journal.HighPriorityMessage` and include all fields that are
printable as strings.
This is archived via a new socket-activated unit listens on
/run/systemd/report/io.systemd.Journal
This commit adds a new OUTPUT_SKIP_UNPRINTABLE to the OutputFlags
and adds code in `update_json_data` and `json_escape` to honor it.
When set all json fields that have unprintable data will be skipped
and `null` is send instead.
We will need a way to send journal entries as metrics. Those are already
json objects. So Lennart suggested to introduce a new type
METRIC_FAMILY_TYPE_OBJECT that does this. This commit implements
his suggestion.
tpm2_index_to_handle() returns 0 with a NULL handle when the NV index is
not present on the TPM. tpm2_nvpcr_extend_bytes() only checked for r < 0,
so a tombstoned NvPCR (anchor file present, NV slot cleared out from under
us) passed the NULL handle to tpm2_extend_nvpcr_nv_index() and aborted the
process via its assert(). Handle r == 0 explicitly, as the other
tpm2_index_to_handle() callers already do.
The newly introduced -ENODEV is mapped together with -ENOENT to the
io.systemd.PCRExtend.NoSuchNvPCR varlink error.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Add tsm_report_acquire(), a thin wrapper around the kernel's
/sys/kernel/config/tsm/report/ configfs interface for fetching a
confidential-computing attestation report (SEV-SNP, TDX, ...), including
a caller supplied input.
A report signing backend that returns a confidential-computing
attestation report obtained via configfs-tsm. Implements
io.systemd.Report.Signer.Sign(): embeds the digest as the report's
inblob and returns the outblob (plus provider and any aux/manifest
blobs). Wired up as the "tsm" mechanism with a socket-activated service.
Add tsm_report_acquire(), a thin wrapper around the kernel's
/sys/kernel/config/tsm/report/ configfs interface for fetching a
confidential-computing attestation report (SEV-SNP, TDX, ...), including
a caller supplied input.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Transfer files might come and go, components might be enabled and
disabled. Patterns might change. Let's keep track of what we install, so
that we can automatically gc everything no longer owned by any enabled
transfer.
Currently they are marked as required, as resolved aborts on startup if
dns-over-tls is built in, even if it is not enabled in the config.
Change initialization to be done lazily on first use, so that if the
config is not enabled, it never runs, and the libraries are never
dlopened, so they can be downgraded to recommends.
At various places we filter directory enumerations by inode type. Let's
add explicit support for that, so that the "struct dirent" array we
return already suppresses them.
This shortens code and makes things more robust.
Without CAP_SYS_ADMIN and without the 'user_subvol_rm_allowed' mount
option, BTRFS_IOC_SNAP_DESTROY is rejected with EPERM (or EROFS for a
read-only subvolume), so rm_rf_subvolume() left subvolumes behind.
test-btrfs thus accumulated leftover subvolumes in /var/tmp on every
unprivileged run on a btrfs filesystem.
An unprivileged owner can however clear the RDONLY flag, empty a
subvolume and rmdir() it. So clear the RDONLY flag on EPERM/EACCES too
(not just EROFS) to leave the subvolume writable, and let rm_rf() fall
through on EPERM/EACCES to empty the subvolume recursively and rmdir()
it, matching what rm_rf_at() already did.
Fixes https://github.com/systemd/systemd/issues/42674
Let's translate openssl's errors to proper errnos
where we can instead of returning EIO for everything.
Let's also make log_openssl_errors() public so we can
use it everywhere and migrate the rest of the codebase
to use it.
Two fixes to get this byte-stable:
- `fd_copy_directory()` was using `FOREACH_DIRENT_ALL`, which doesn't
give stable ordering. Read all paths, sort, then iterate.
- `mcopy -s` depends on `readdir()` ordering and thus isn't
reproducible. Implement the recursion/sorting here and only invoke
mcopy/mmd per dir.
First change increases memory usage, as we don't stream the paths
anymore, second increases the number of context switches when invoking
external tools. Both should be fine given the ESP content should usually
be pretty limited.
I'd like to write a test for this, but didn't come up with a way that
doesn't require privileges and would surface the error reliably.
The TPM2_NV_Read commands returns the requested data in a
TPM2B_MAX_NV_BUFFER type, the maximum size of which is TPM-specific and
can be determined by querying the value of the TPM_PT_NV_BUFFER_MAX
property.
The value of this may be smaller than the payload size of some NV
indexes, particularly when that payload is a X509 certificate with a RSA
public key. Eg, the manufacturer supplied RSA EK certificate on my own
machine has a size of 1035 bytes, and the value of TPM_PT_NV_BUFFER_MAX
is 1024.
To handle this case and make it possible to read any EK certificate from
the TPM, make tpm2_read_nv_index support chunked reads when the payload
size is larger than what the TPM can return in a single command.
OpenSSL 4 broke ABI, so we need to look for both SONAMEs.
Try libssl.so.3 first, and fallback to libssl.so.4,
so that the older and more stable version is used if both
are installed, giving distros time to fix regressions.
Follow-up for ccdd42351f
Fixes https://github.com/systemd/systemd/issues/42675
During soft-reboot teardown, systemd-cryptsetup@.service's ExecStop runs
libcryptsetup's crypt_deactivate(), which issues DM_REMOVE with a udev
cookie and blocks in dm_udev_wait() until 95-dm-notify.rules decrements
the cookie semaphore via "dmsetup udevcomplete $env{DM_COOKIE}".
The generated cryptsetup unit had only After=systemd-udevd-kernel.socket,
not After=systemd-udevd.service. The .socket has IgnoreOnIsolate=yes and
is not stopped during soft-reboot, so it provides no ordering at all for
the service teardown. Meanwhile systemd-udevd.service has
Conflicts=soft-reboot.target (added in 0d1819e791) and stops as soon as
soft-reboot.target starts, with no ordering relative to cryptsetup's
ExecStop.
Once udevd's device monitor event source is disabled in manager_exit(),
pending DM_REMOVE uevents are no longer processed and the cookie
semaphore stays at 1 forever, blocking soft-reboot at "Stopping
Cryptography Setup..." until JobTimeoutSec=30min on soft-reboot.target
fires.
Add systemd-udevd.service to the After= ordering of the generated
cryptsetup, veritysetup and integritysetup units. By systemd's job
ordering rules (job_compare() in src/core/job.c), when two units are
both being stopped and one has After= the other, the After= unit is
stopped first. So with cryptsetup@*.service After=systemd-udevd.service,
cryptsetup stops first (cookie acknowledged by the still-live udevd),
then udevd stops.
Putting After=umount.target on systemd-udevd.service does not work: at
soft-reboot, udevd's stop job runs concurrently with umount.target's
start job, and JOB_STOP unconditionally precedes JOB_START in the
transaction (see src/core/job.c:1742). The ordering has to be expressed
between two stop jobs, which is what putting After=systemd-udevd.service
on the dm consumers achieves.
Fixes: #40298
The existing tpm2_index_to_handle helper has optional return arguments
for an object's public area and its qualified name. However, it can also
be called for handles corresponding to NV indexes, where the public area
and qualified name don't apply.
I'm working on another change that would benefit from having an
equivalent helper for NV indexes, and there is already some code in
shared/tpm2-util.c that could use this as well.
This splits the existing tpm2_index_to_handle function into 3 functions:
- tpm2_index_to_handle, which works for both objects and NV indexes and
returns a name and handle.
- tpm2_object_index_to_handle, which works the same as the existing
function for transient or persistent objects, returning a public area,
name, qualified name and handle.
- tpm2_nv_index_to_handle, which works for NV indexes, returning the NV
public area, name and handle.
This commit adds support to /etc/hostname for substitution of $ from
wordlists located in /etc,/run,/usr/lib}/systemd/. Each $ is resolved to
a number (1,2,3...) and the corresponding file "1" is opened to acquire
the word. With that we can do a petname [1] style hostname in systemd,
e.g. below a possible expansion for a hostname template:
$-$-$-???? -> wildly-happy-octopus-92a9
The substitution of words is stable (based on machine-id) but if the
wordlist changes the hostname would change. We could pick it once and
cache it but Lennart did not like this so this version instead always
picks it (based on offset of the file so the operation is cheap). To
persist it one can use the `firstboot.hostname` credential. One a live
system this will be expanded and then written in the expanded form to
/etc/hostname.
This also includes a wordlist from the "petname" project that can be
optionally installed.
Thanks to Dustin Kirkland for this wonderful project.
[1] https://github.com/dustinkirkland/petname
---
I'm a bit unsure if this should include the word lists (I think its nice
to have them though) and if so if they should be their own commit.
`tpm2_get_best_pcr_bank()` only ever considered the SHA256 and SHA1
banks (both the `LoaderTpm2ActivePcrBanks` path and the capability
guesswork). On a TPM whose only active bank is SHA384 it returned
`-EOPNOTSUPP`, breaking sealing/enrollment (cryptenroll, credential
encryption, legacy unseal). The restriction looks like a historical
simplification — `efi_get_active_pcr_banks()` already decodes
SHA384/SHA512 and `tpm2_hash_algorithms[]` already lists them.
This PR introduces an explicit preference table (SHA256 > SHA512 >
SHA384 > SHA1) and selects from it. SHA256 stays the top preference for
backwards compatibility, so existing systems keep using the same bank
and the legacy unseal-guess in `tpm2_unseal()` stays consistent;
SHA384/SHA512 are only chosen when SHA256 is unavailable, SHA1 remains
the last resort.
Behavior for existing SHA256/SHA1 systems is unchanged. Includes a unit
test for the bank-preference logic.
Related to https://github.com/systemd/systemd/pull/42537
Lennart suggested to use a more uniform algorithm for
the picking of the hostname words that is not biased
for long words by just (predictably) randomly going over
the offsets until we land on a word boundary. This is a
very nice suggestion so this commit implements it with
a fallback to the "old" behavior if we do not find a
word boundary within a reasonable amount of attempts.
A small python script shows that 64 iterations plus
fallback is a good number:
```
$ python3 simulate-hostname-pick.py 64
hostname-wordlist/adverbs
words=261 p_accept=0.1119 avg_bytes/word=1/p=8.94
max_iterations=64, n_trials=1000000
fallback rate : 0.051000% (510/1_000_000)
mean seeks per word : 8.93
hostname-wordlist/adjectives
words=449 p_accept=0.1380 avg_bytes/word=1/p=7.24
max_iterations=64, n_trials=1000000
fallback rate : 0.007500% (75/1_000_000)
mean seeks per word : 7.25
hostname-wordlist/nouns
words=449 p_accept=0.1472 avg_bytes/word=1/p=6.79
max_iterations=64, n_trials=1000000
fallback rate : 0.002700% (27/1_000_000)
mean seeks per word : 6.79
```
Combined with the fallback to the previous method if
we can't find anything within the 64 attemps this seems
to be the best tradeoff and give us very good uniformity.
Similar to the support for ?/$ in /etc/hostname and the credentials
we now add this to the kernel commandline systemd.hostname= option.
If the expansion fails, e.g. in the initrd where the word lists (and
possibly the machine ID) are not available yet, the option is ignored
and the usual default hostname logic applies. Once the host system is
up the expansion succeeds and the intended name is applied.
Now that we support the `$` we want to also make this available
inside the system.hostname and firstboot.hostname credentials and
the firstboot --hostname option. This commit adds it (and also `?`).
This commit adds support to /etc/hostname for substitution
of $ wordlists from {/etc,/run,/usr/lib}/systemd/hostname-wordlist.
The first $ will lookup hostname-wordlist/1, the next
hostname-wordlist/2 and so on.
With that we can do a petname [1] style hostname in systemd, e.g.
below a possible expansion for a hostname template:
$-$-$-???? -> wildly-happy-octopus-92a9
The substitution of words is stable (based on machine-id) but
not persisted, it is picked on every boot via a stable file
offset so the operation is cheap. But this means that if the
wordlist changes the hostname would change. The next commit
will add the pattern to the firstboot.hostname credential which
is persistet with the resolved names to avoid this issue.
This also includes a wordlist from the "petname" project
that can be optionally installed.
Thanks to Dustin Kirkland for this wonderful project.
[1] https://github.com/dustinkirkland/petname
For a good user experience, users expect to be informed of how many
attempts they have before being locked out of their FIDO2 device.
By displaying such information in advance, the user can make strategy to
obtain the accurate PIN or wait when they are close to an authority who
can provide them for a recovery key.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
We already support exponential backoff for automatic restarts via
RestartSec=/RestartSteps=/RestartMaxDelaySec=, but there is no way to
randomize the restart delay. When many instances of a service fail at
the same time (e.g. because a shared resource briefly went away) they
are all restarted in lockstep, creating a thundering herd problem.
So this commit adds a simple `RestartRandomizedDelaySec=` service
option which is similar to the timer `RandomizedDelaySec=` and
adds a randomized restart delay.
Fedora has switched to openssl 4, and we generate a Requires dependency
on libcrypto for the systemd-udev subpackage, so preferring openssl-3
does the wrong thing. So the order in the dlopen note needs to be switched.
But in general, we want to get rid of openssl-3, so we want to load
openssl-4 in preference. Change the order in both places.
(The "compat" order can stay in 261-stable for other distros.)
This PR fixes userdb lookups for NSS users that are resolved through an
alias but returned with a canonical user name.
Some NSS providers, such as SSSD, can successfully resolve a user by an
alias-like name, for example a Kerberos/AD UPN (for example
testuser@example.test) while returning a passwd record with the
canonical login name.
The original lookup name was not preserved. Later, the userdb worker
checked whether the returned record matched the requested name with
user_record_matches_user_name(). Since the requested name was
testuser@example.test, but the record only contained testuser, the
lookup was rejected as:
`io.systemd.UserDatabase.ConflictingRecordFound`
This also caused pam_systemd to fail opening sessions for such users
with:
`pam_systemd(...:session): Failed to get user record`
In my case, this broke graphical logins for Samba users logging in with
a UPN on systems where SSSD canonicalizes the NSS result.
My solution preserves the requested name as a UserRecord alias when an
NSS lookup by name succeeds but the returned pw_name differs from the
requested name.
This allows user_record_matches_user_name() to accept the canonicalized
NSS result instead of treating it as a conflicting record.
The patch does not invent new aliases. It only records the name that NSS
itself already accepted and resolved successfully.
I'm not an expert on working with systemd, so I'm asking for a review of
my PR.
When multiple swap devices exist, prefer one with enough free space
to hold the hibernation image over one that cannot, regardless of
priority. If no swap can fit, fall back to priority-first selection.
This avoids deterministically failing hibernation when the
highest-priority swap is too small but a lower-priority one fits.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Co-developed-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This adds the following:
1. systemd-report gains a new --sign= option, taking a boolean. If true,
this makes systemd-report generate + systemd-report upload generate a
signed report, instead of a regular one. The signatures are collected
from Varlink-based backends.
2. One such backend is added which does a simple Ed21159 based signing
scheme.
3. this adds a new metrics source which just reports text files
symlinked into a special dir as metrics. This is used to report the
Ed21159 public key as metric, by default, if it exists.
4. finally, systemd-report itself is turned into a varlink service. this
is useful for example for extracting a report from a system coming in
via the varlink/http bridge.
I thought a long time about the format of signing of reports. Initially
i intended to do this like homed's user record signing, i.e. require
normalization of the record, then normalize the record, and write it out
in dense form, since the result. Finally insert the resulting hash into
the user record itself. People have pointed me to the inherent messiness
of signing JSON this way though, as it requires any participant that
wishes to sign/authenticate records this way to implement the exact same
normalization/formatting rules, and in particular in the area of
floating point numbers (of which metrics presumably will have many) this
is quite problematic.
This signing hence goes a different way. instead of expecting
signer+verifier to independently come to the same normalized text form
of the json data, let's instead output a JSON-SEQ sequence, where the
first object is the report, and any subsequent objects are one signature
each. the signatures are supposed to cover the precise binary
representation of the first element in the JSON-SEQ stream. (i.e. from
the RS to the NL).
or in other words: a verifier would receive the JSON-SEQ stream, split
it up before each RS. Then it would leave object 1 unparsed for the
moment, and parse objects 2…n. It would then authenticate object 1's
precise binary representation with objects 2…n. Once that checks out, it
would parse object 1, and use it as report.
Acquiring a LUO session from /dev/liveupdate requires privileges, and
also the device is a single-owner driver so only a single process can
open it at any given time.
Add a LUOSession= service settings that allows units running without
privileges to get a session assigned to them.
The kernel imposes a 64 chars limit on session names, which is too short
to avoid clashes, so derive a hash from joining the unit name with the
parameter name, that way two units using the same setting don't clash.
OpenSSL 4 broke ABI, so we need to look for both SONAMEs.
Try libcrypto.so.3 first, and fallback to libcrypto.so.4,
so that the older and more stable version is used if both
are installed, giving distros time to fix regressions.
tpm2_get_best_pcr_bank() only ever considered SHA256 and SHA1, both in
the firmware-reported LoaderTpm2ActivePcrBanks path and in the
capability-based guesswork. On a TPM whose only active bank is SHA384 it
returned -EOPNOTSUPP, which for units ordered before sysinit.target
(systemd-pcrextend/systemd-pcrproduct.service) propagates and hangs the
boot, even with --graceful (that only covers "no TPM present").
Introduce a preference table (SHA256 > SHA384 > SHA512 > SHA1) and select
from it. SHA256 stays top for backwards compatibility; SHA384 is
preferred over SHA512 as it produces a shorter digest and so uses less of
the bounded TPM event log; SHA1 is the last resort. SM3/SHA3 are excluded
as we cannot compute those digests in software.
The legacy unseal path in tpm2_unseal() re-derives the bank for old
enrollments that did not record one (pcr_bank == UINT16_MAX). Those were
sealed by code that only picked SHA256 or SHA1, so the wider preference
could now re-derive a stronger bank the secret was never bound to and
silently fail to unseal. To stay compatible, the selection takes the
preference list as a parameter, and a _legacy() variant restricted to
{SHA256, SHA1} is used for that path; fresh-sealing callers keep the full
preference.
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
These signatures can be provided by arbitrary Varlink backends symlinked
into /run/systemd/report/sign.
Also add a proper Varlink IDL for the upload interface, since we are
extended it now.
Acquiring a LUO session from /dev/liveupdate requires privileges,
and also the device is a single-owner driver so only a single
process can open it at any given time.
Add a LUOSession= service settings that allows units running
without privileges to get a session assigned to them.
The kernel imposes a 64 chars limit on session names, which is
too short to avoid clashes, so derive a hash from joining the
unit name with the parameter name, that way two units using
the same setting don't clash.
We might want to add more state to the LUO session json payload,
so add a version (to allow clean compat breaks if needed) and nest
the current fdstore contents under a 'units' object, so that more
top-level data can be added in the future without breaking
backward compatibility.
Follow-up for 257c35c1a3
663f0bf5cb stopped reusing the original block device fd whenever
partition scanning was requested (LO_FLAGS_PARTSCAN) but couldn't be
enabled on the device, so that nested partition tables on devices the
kernel won't scan (e.g. the pmOS/android case) get exposed via a real
loop device.
However that also forced a pointless loop device for any partition that
carries a file system directly, e.g. a btrfs subvolume mounted via
MountImages=. For multi-device btrfs this is fatal: the kernel rejects
seeing the same member via both the original partition and the loop
device, and the mount fails.
A loop device is only ever needed here to expose a nested partition
table. So only refuse the shortcut when the device actually carries one,
probed via gpt_probe(), instead of whenever partition scanning is
disabled. Devices carrying a file system directly (or nothing) take the
shortcut as before.
Add an integration test to cover the failure scenario of the original
issue.
Fixes: https://github.com/systemd/systemd/issues/42520
Replaces: https://github.com/systemd/systemd/pull/42576
Follow-up for 663f0bf5cb79ecaf6dd71441ecdc9dc401e7eae6
Co-Authored-By: Luca Boccassi <luca.boccassi@gmail.com>
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
NSS lookups may resolve an alias-like name to a canonical user or group
name. For example, SSSD can resolve a userPrincipalName such as
"testuser@example.test" while returning a passwd record whose pw_name is
the canonical login name "testuser".
The NSS userdb worker currently converts such records using only the
canonical name. As a result, the originally requested lookup name is lost,
and lookups through the varlink/multiplexer path may fail with
ConflictingRecordFound even though direct NSS lookup succeeds.
Preserve the originally requested lookup name as an alias when it differs
from the canonical passwd/group name and passes normal alias validation.
Add the alias both to the parsed record fields and to the JSON
representation so serialized records remain self-consistent.
Extend group record loading and matching to understand aliases too.
When both a name and UID/GID are specified in a userdb lookup, prefer
the by-name path so alias-aware records can satisfy the subsequent
consistency checks.
This allows canonicalized NSS results to match the original lookup name
correctly and fixes session setup for users logging in with
NSS-resolvable aliases such as UPNs.
do_mcopy() invoked 'mcopy -s' with the top-level sources sorted by name,
but then relied on mtools' built-in recursion to walk into subdirectories.
mtools uses platform readdir(), so the order in which it inserts entries
into the freshly-formatted FAT image inherits whatever order the source
filesystem returns. With an ext4 staging directory the directory hash
changes the readdir order between hosts (and between runs after edits),
and because FAT stores directory entries in insertion order the resulting
ESP differs byte-for-byte across otherwise identical builds.
Stop using 'mcopy -s'. Walk the source ourselves with
readdir_all_at(RECURSE_DIR_SORT) at every level and emit per-item mtools
commands interleaved per parent: each subdirectory is created with 'mmd'
before the recursion descends into it, and consecutive sibling files are
batched into a single 'mcopy' invocation so the fork cost stays
proportional to the number of directory transitions rather than to the
total number of files.
Combined with the existing 'mkfs.fat --invariant' and SOURCE_DATE_EPOCH
plumbing, this makes vfat populated via repart byte-stable regardless of
the staging filesystem's readdir() behaviour.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Lennart Poettering
Michael Vogt
Paul Meyer
Luca Boccassi
Daan De Meyer
Chris Coulson
Rocker Zhang
Raito Bezarius
Zbigniew Jędrzejewski-Szmek
dongshengyuan
Ivan Kruglov
Arseniy Kostevich
Chitoku