This time, various commands with non-trivial requirements are handled.
So the series is shorter, but has a pattern of feature additions and
then those new features being used. The way that --help output is
formatted is changed a bit in various places. Those changes are
described in individual commits.
22 'static int help()' functions left after this.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added.
Options are now in two groups: "Common options" and Mount options". In
--help, for systemd-mount, both groups are shown. For systemd-umount, only
the first. This changes the behaviour a bit: previously, both groups were
shown together as one list. I think the new behaviour is actually a bit
better, because it makes it easier for the user to figure out that the
commands share common options and one of the commands has its private
second group. But if people think that the previous behaviour should
be restored, we could add a display options to squish the groups back
together.
--help now goes through the pager, like other commands with --no-pager.
So far, option groups were only used for display: when parsing options,
the group markers were ignored and all options within a namespace were
considered as potential matches. The group was reported in introspection
data and used as section seperator in help display. To allow a single
option namespace to be used for two commands, pipe through a nulstr with
option group names to look at when parsing options. The field is set in
the Command description and used in various option-handling contexts:
- when parsing and looking for an option, do not consider the entries
that are in the group that is not listed in the nulstr,
- when listing options in help or introspection, skip over unlisted
groups.
The verbs_find_verb and verbs_find_command helpers are adjusted to have
a consistent interface.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. --help now goes through the pager, like other
commands with --no-pager.
Also, previously, -h/--help/--version were listed in the "Commands"
section. This was cute, but recreating this in the new scheme would
be complicated. Also, it seems unnecessary. --help/--version and some
other commands terminate the program in all binaries, so they could
be considered "commands" in all cases. But we printed them together
with verbs only in a few cases. So I think it's fine to move them
back to the "Options" section. This makes the description of options
much more consistent between different programs.
This allows commands like systemd-sysext + systemd-confext to be handled
without duplicating entries in the verbs table. The basic idea is to have
the following layout in the SYSTEMD_VERBS section:
VERB_COMMAND_MARKER → "systemd-sysext", .flags = COMMAND_VERBS_SHARED
VERB_COMMAND_MARKER → "systemd-confext", .flags = COMMAND_VERBS_SHARED
VERB("status"),
VERB("merge"),
...
And then in places where we want to work on the verbs, we skip over the
verb entries with VERB_COMMAND_MARKER and COMMAND_VERBS_SHARED flag set.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The note about invocation as 'installkernel' is
kept as the footer. Verb and option column widths are kept separate, since
the verb table is very wide. --help now goes through the pager, like other
commands with --no-pager.
A separate COMMAND block is included for installkernel, but it is only
used for introspection data.
The custom halt_help() and shutdown_help() are replaced by COMMAND()
descriptions (one per command name, halt/poweroff/reboot sharing the
"halt" option namespace), and --introspect-cli is added to both parsers.
halt/poweroff/reboot now each reference their own man page name.
journal_file_open_reliably() recovers from nine distinct errors, but the
warning it emits names none of them:
systemd-journald[47980]: File /var/log/journal/95790226b8d0779f4b4797314ca986d4/user-7736.journal corrupted or uncleanly shut down, renaming and replacing.
So an administrator cannot tell an unclean journald shutdown (-EBUSY)
from real corruption (-EBADMSG), an incompatible feature flag
(-EPROTONOSUPPORT) or a failing disk (-EIO) — all of which reach this
line. The reason is known at this point, but only reachable through the
log_debug_errno() calls in journal_file_verify_header(), which are off by
default and gone by the time the warning is noticed.
Append %m so the error is part of the message. It now reads
File /run/log/journal/…/user-7736.journal corrupted or uncleanly shut down, renaming and replacing: Device or resource busy
for a file left online by an unclean journald exit, and
… renaming and replacing: No data available
for a truncated one.
Image builds create a key separate from the SecureBoot one for verity
signing. Its certificate ships inside the image as the sysext trust
anchor.
`genkey` certs had a hardcoded "SecureBoot signing key on host <fqdn of
the build machine>" CN, which describes neither that verity role nor a
meaningful hostname.
Make that name configurable with `SecureBootCertificateCommonName=`,
like mkosi's `--genkey-common-name`. That allows image build tools to
re-use `ukify genkey` for non-SB keys.
Unlike the implicit truncation for the default name, a specified one
which is too long is an error. Add a citation where that limit comes
frm.
This adds FONT_SCALE= setting in vconsole.conf,
vconsole.font_scale= kernel command line option and credential support.
When 2 or 200% is specified, the console font size will be doubled.
Closes#43273.
Create the file with open_tmpfile_linkable_at().
Write the recovery key and trailing newline to the temporary fd.
Set the final mode to 0600 before publishing it.
Link it into place with link_tmpfile_at(..., LINK_TMPFILE_SYNC).
In review it was raised that the file is often synthesized and that this
synthetization might not include all the fields, let's allow the more
recently added fields to be missing to avoid a regression in
compatibility.
As discussed in the PR, those fields can go down, so they cannot be advertised
as type COUNTER. But the type is shared for the whole family, so we need to
create a new family to keep the other ones as COUNTER and add the new ones
as GAUGE.
kernel/sched/cputime.c says above get_cpu_idle_time_us() and get_cpu_iowait_time_us():
> Note that this is partially broken due to the counter of iowait tasks that
> can be remotely updated without any synchronization. Therefore it is possible
> to observe backward values within two consecutive reads.
steal is apparently also unreliable because it is reported by the hyperviser.
So let's be cautious and report it as gauge too.
Extend the io.systemd.Basic metric families with host-level runtime
metrics: cumulative per-state CPU seconds, memory in use (MemTotal minus
MemAvailable), swap in use, the kernel's 10s pressure stall averages
(cpu/memory/io x some/full). Partitions and stacked devices are excluded
from the disk counters, since their traffic is already accounted for in
the parent device.
Also extend the TEST-74 report subtest with checks for the new metrics.
Network metrics are not added, because the service runs with
PrivateNetwork=yes, and it doesn't make sense to report stats for 'lo'.
After some back-and-forth, I settled on "CPUUsage". We use this scheme of
capitalization in other metric family names: "IPv4Public", "IPv6Public",
IPv4AddressState", "IPv6AddressState", "OSRelease" "SMBIOS" "BootID",
"CPUsOnline", "MachineID", "MachineInfo", "OSRelease",
"SMBIOS.SystemManufacturer", "TPM2.Manufacturer", "TPM2.VendorString",
"IOReadBytes", "IOReadOperations", "CGroup". The exception is
"io.systemd.cgroup.CpuUsage". I think we should treat that one as a
mistake and consider changing it to conform.
Most of the time, the CLI tools that will use this helper are dealing
with the string representation of log levels, so just add string-based
helpers for now.
We already have io.systemd.service.SetLogLevel. Add a getter so that
CLI tools with `log-level` subcommands can be more easily implemented
with varlink.
Bind this method to all services that already bind io.systemd.service.SetLogLevel.
When we set up a device with LUKS or verity, we so far didn't release
references to the allocated device nodes even though we destructed the
volumes they are for. Let's fix that, and release everything.
This mostly matters for cases where multiple attempts to unlock are made
(for example because of a password issue), without releasing the
DissectedImage.
Serving stale cache data on a retry is a resilience feature for unicast
DNS: when the configured server does not respond, an expired answer beats
no answer. The gate in dns_transaction_go() that switches the cache lookup
into serve-stale mode from the second attempt on only checks
StaleRetentionSec=, not the scope's protocol, so it applies to mDNS and
LLMNR scopes as well. There it is wrong twice over:
* With StaleRetentionSec= set, a query for a peer that has left the
network misses the cache on the first attempt (the record's TTL has
expired), gets no reply, and is then answered from the expired but
retained record on the retry — for as long as the retention window
lasts. On the link-local protocols a retry means that no peer
answered, not that some server is unreachable, and RFC 6762 treats
TTL expiry as a presence signal: a host whose records lapse is gone.
* Independently of whether anything stale is cached, a cache hit made
in serve-stale mode is reported with its TTL clamped to 30s (see
dns_cache_lookup()) and counted in the served-stale statistics. mDNS
transactions are retried at 1s, 2s, 4s, ... whenever nobody answers,
so a record announced or learnt from another transaction in between
is handed to the client with a 30s TTL instead of its real remaining
TTL, and logged as a "Serve Stale response" although it never was
stale.
Hence only enter serve-stale mode on unicast DNS scopes. This is the
lookup-side counterpart of the cache-side change that stops link-local
records from being retained past their TTL at all.
`STOPPING=1` is final, explain in the sd_notify man page that this is
the case and log an error if a service attempts to get back out of
STOPPING with another notification.
Fixes#43465 / 98734ac74d
This drops a good chunk of 21ab2fe6ec
again, which I think is plainly wrong or at the very least too
overengineered.
1. it's wrong to assume that overlayfs layers are mounted elsewhere in
the fs. it's quite likely they aren't. and overlayfs correctly passes
through fsync() to the top layer, hence we really should keep syncing
it.
2. network file systems deserve syncing. This code here runs in the
initrd, where a very restrictive set of file systems is mounted and
which runs for a very short time only, i.e. the circumstances haven't
really changed. Hence I am very sure it's better to sync file systems
here than not to, we do this for safety reasons after all.
Reliability always trumps speed here, we should not give up
correctness over presumed speed improvements.
3. There's zero reason to believe FUSE doesn't need syncing. We live in
a world where increasingly complex kernel file systems are moved to
userspace, and hence FUSE can be used to implement *anything*. Hence
we should sync it. Yes, FUSE means some backing process needs to run,
but that's a job for the initrd to get right: make sure the process
runs properly over the switch root if the mount is left in tact over
the switch root too.
Hence, dumb this all down and drop the weird fine tuning here.
In practice I think this will ever only be used with ansi_highlight_red(),
but it's cleaner to add this generic extension rather than e.g. a boolean
highlight_footer_red.
The options is passed through via command_print_help_full(), not through
the COMMAND structure, which means that commands with 'help' verb are not
covered by this. The red footer highligting is used for legacy commands,
which don't have verbs, so I think this is OK. Making it generic and
covering 'help' verbs would require more refactoring, and I think it
doesn't make sense to add it until it's actually needed.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The generic help-verb check in the test is
filtered out, since systemctl's "help" verb shows unit manuals and fails
without arguments.
Almost all programs are simple and have only a single command. When
printing options, let's just use the first COMMAND block found by default.
This means we can drop the explicit program-name argument and reduce
copy&paste boilerplate.
Otherwise, random input cases are not tested:
```
/* test_compress_decompress_blob */
/* testing xz text blob compression/decompression */
Loaded shared library 'liblzma.so.5' via dlopen().
/* testing xz random blob compression/decompression */
compression failed: No buffer space available
/* testing lz4 text blob compression/decompression */
Loaded shared library 'liblz4.so.1' via dlopen().
/* testing lz4 random blob compression/decompression */
compression failed: No buffer space available
/* testing zstd text blob compression/decompression */
Loaded shared library 'libzstd.so.1' via dlopen().
/* testing zstd random blob compression/decompression */
compression failed: No buffer space available
/* testing gzip text blob compression/decompression */
Loaded shared library 'libz.so.1' via dlopen().
/* testing gzip random blob compression/decompression */
compression failed: No buffer space available
/* testing bzip2 text blob compression/decompression */
Loaded shared library 'libbz2.so.1' via dlopen().
/* testing bzip2 random blob compression/decompression */
compression failed: No buffer space available
```
StaleRetentionSec= is a resilience feature for unicast DNS: when the
configured servers are unreachable, answering from an expired cache
entry beats answering nothing at all. That rationale does not carry
over to the link-local protocols, where there is no configured server
that could be unreachable -- peers simply come and go, and a record's
TTL is the only thing bounding how long a departed peer is still
answered for.
For mDNS the mismatch goes further. RFC 6762 treats record expiry as a
presence signal -- a host or service whose records lapse is to be
considered gone (section 10.1 goodbye packets even force this early by
publishing TTL=0), and the section 5.2 cache maintenance scheme exists
precisely so that expiry happens on time. Retaining expired mDNS
records for StaleRetentionSec= contradicts that: vanished peers linger
in the cache for the whole retention window, are served to clients as
if still present, and DNS-SD browsers do not see their services
disappear when the records' TTL lapses.
Beyond lingering entries, retention also swallows goodbyes outright, as
noted in review of the goodbye re-arm fix [1]: dns_cache_prune() and
dns_cache_expiry_in_one_second() key on the retention-extended "until",
so with StaleRetentionSec= set a goodbye's TTL=1 rewrite lands a full
retention window out -- the one-second prune of RFC 6762 section 10.1
then drops nothing, nothing re-arms, and the withdrawn service stays
listed for the whole window.
LLMNR has no goodbye mechanism, so only the lingering half applies to
it, but it does apply: dns_transaction_cache_answer() passes the
retention through for LLMNR replies just like for unicast DNS, and the
stale-serving gate in the transaction code does not look at the
protocol, so a departed LLMNR peer keeps being answered from cache for
the whole window as well.
Zero the retention centrally in dns_cache_put() for everything but
unicast DNS, which covers both put paths: transaction replies and the
mDNS packets cached directly by on_mdns_packet(). Add tests pinning
both directions, including the goodbye rewrite of an existing mDNS
entry.
[1] https://github.com/systemd/systemd/pull/43259#discussion_r3809488076
The DBus API already exposes the parsed machine-tags list via the Tags
property but varlink clients needed to parse the raw TAGS= line from
MachineInformationData (which also required implementing tag name
validation if the tags would later be used with SetTags, to avoid
spurious errors).
So, provide the same information that DBus gets (a list of pre-parsed
tags with bad tags gracefully removed) as a new MachineTags field
returned by Describe -- this does mean that the DBus API now exposes the
same data in two ways but making Describe() consistent is more important
and mirrors the OperatingSystem* class of properties.
I had a situation where Dell Wyse 3040 with SK Hynix eMMC stopped booting
after upgrading from Fedora 42 to Fedora 44 with systemd 259.8.
The card reports a product name containing byte 0x92, but the kernel event
also contains the valid module alias needed to load mmc_block:
MMC_NAME=H8G4a\222
MODALIAS=mmc:block
I inserted the mmc_block kernel module manually and booted successfully.
Then after enabling udevd debug logging and retriggering the MMC add event
via
udevadm control -l debug
udevadm trigger -v -c add -s mmc
udevadm settle
I saw the failure in the "journalctl -b -u systemd-udevd.service"
output:
mmc0:0001: sd-device: Refusing invalid property: MMC_NAME=H8G4a\222
mmc0:0001: sd-device: Failed to add property 'MMC_NAME=H8G4a<92>': Invalid argument
sd-device-monitor(manager): Failed to create device from received message: Invalid argument
Since PR 41001, strict property validation returns EINVAL for the bad
MMC_NAME. device_new_from_nulstr() propagated that error and discarded
the whole event, so udev never processed MODALIAS and did not load
mmc_block.
Skip ordinary properties whose keys or values fail validation. Do not store
them, but continue processing the remaining properties. Required and typed
properties are handled separately and remain fatal when invalid, as do
malformed records and all other errors.
Closes: https://github.com/systemd/systemd/issues/42930
Related: https://github.com/systemd/systemd/issues/41339
Related: https://github.com/systemd/systemd/issues/43008
Similar: https://github.com/systemd/systemd/pull/42316
Similar: https://github.com/systemd/systemd/pull/43061
Regression: https://github.com/systemd/systemd/pull/41001
image_rename() updated the Image object name and path before
computing the source paths for auxiliary files. As a result, the
auxiliary rename loop looked for files with the new image name,
while existing sidecars such as .roothash, .verity, and
.raw.tpmstate still used the old image name.
Delay updating the Image object until after renaming its settings
and auxiliary files. This lets image_auxiliary_path() resolve
existing sidecars from the old image name while
rename_auxiliary_file() assigns the new name.
Extend the import test raw image rename coverage to check
representative sidecars are renamed together with the image.
nsresourced hands out transient, recyclable UID/GID ranges to
unprivileged user namespaces, and a BPF-LSM policy is what keeps a
client from leaving inodes owned by its transient range on a file system
that outlives the namespace, so that a later client which gets the same
range can't inherit them.
The old policy decided based on the mount a write went through, which is
forgeable: unshare(CLONE_NEWNS) gives you clones of every mount with
fresh mount IDs but the same superblock, and the mount namespace's owner
is yours to pick. overlayfs sidesteps it entirely. So this reworks the
check to look at where the write actually lands on disk.
The first commit (Christian Brauner's) replaces the per-mount allowlist
with two questions answered from fields the client can't forge and that
survive mount cloning: does the file system die together with the user
namespace (sb->s_user_ns at or below the managed one), and does the
transient ID actually reach the disk, or does an idmapped mount
translate it into something else first. It walks the userns chain
upwards so a nested namespace can't escape it. The map layout changed,
so the pins move to userns-restrict.v2 and the old ones are removed on
install.
The second commit handles overlayfs. A write through an overlay creates
the file on the upper layer via vfs_create(), which only trips the inode
LSM hooks, not the path hooks we attach to, and the one path hook that
does fire sees the overlay mount, which belongs to the managed namespace
and so looks like it dies with it even though the upper may be
persistent. So we judge the upper at mount time in sb_kern_mount and
refuse a writable overlay unless its upper dies with the namespace too.
Reading overlayfs' private ovl_fs/ovl_layer uses CO-RE type flavors
guarded by bpf_core_type_exists(), and the service now orders after
modprobe@overlay.service.
The third commit deals with restarts. The managed set only lives in the
pinned maps, so an upgrade that changes the map layout comes up with
empty maps and every already-running namespace silently loses its
restriction. manager_startup() now re-adds the live registry entries to
the maps in the same pass that reaps the dead ones. The setgroups()
denial is restored from a flag we now store in the registry.
One consequence: a managed namespace can only mount a writable overlay
whose upper dies with it, e.g. a tmpfs it set up itself. If you need a
writable persistent upperdir (mkosi does), use a plain unprivileged user
namespace for that, not a transient nsresourced range.
UnsetEnvironment= was applied to the environment block before the
command line was expanded. Since the expansion looks the variables up
in that same block, any variable listed in UnsetEnvironment= was
already gone when $VAR in ExecStart= and friends was expanded, and a
whole-word $VAR reference was dropped from the command line.
Let the expansion see the full environment block, and apply the
unsetting only to the environment block passed to the executed
process. This way UnsetEnvironment= can hide variables from the
executed process without breaking command lines that reference them.
Extend the exec-unsetenvironment test accordingly: the whole-word
$VAR6 argument must be expanded into a positional parameter for the
shell, while ${VAR6-X} must evaluate to X in the executed process.
Also document in both the UnsetEnvironment= description and the
"Environment Variables in Spawned Processes" section that the
unsetting happens after command line variable expansion.
Fixes: #43343
When the manager tears down every unit during reload or shutdown,
unit_free() removes each dependency from both endpoints. This work is
redundant because every dependency map is about to disappear, and it
scales poorly with thousands of synthesized mount units.
Clear all unit dependency maps in one manager-wide pass before invoking
the existing unit_free() cleanup.
With 10,000 bind mounts, this reduced the mean daemon-reload time from
63.558s to 14.708s.
Yu Watanabe
Zbigniew Jędrzejewski-Szmek
Paul Menzel
Martin Pitt
dongshengyuan
Nick Rosbrook
Lennart Poettering
Christian Glombek
David 'equinox' Lamparter
Aleksa Sarai
Jakub Stasiak
lucia-huang7
Luca Boccassi
tunaichao
Chris Ayoub