Let's make it possible to "suggest" that certain features or components
are enabled under some conditions.
For this, both features and components gain two things:
1. A Suggested= field which takes a boolean. If true the
feature/component will be suggested for installation, if false it
will not.
2. A set of SuggestedOnXYZ= settings are modelled after ConditionXYZ= in
unit files (and implement a subset of them), will suggest some
component/feature under specific conditions.
The result of the condition is shown in the various output tools.
These are philosophically similar concepts: one deletes old versions
based on whether they are old, and the other deletes old files based on
whether they are orphaned, let's list them together.
These do what updatectl's verbs of the same name does, but are
implemented as low-level concepts in sysupdate itself.
(The idea is to eventually wrap this in Varlink IPC, and make updatectl
use them that way.)
Let's switch to string_is_safe(), and make this available to the rest of
the sysupdate sources too.
This both relaxes and tighten the rules slightly. i.e. control character
and stuff are no longer allowed, but valid UTF-8 (as opposed to ASCII)
now is.
This is currently effectively not supported (as we sometimes prefix the
definitions path with root and sometimes not), let's make this official
for now and refuse it. We could in theory support the combination but I
don't see the big benefit of it for now, hence let's just refuse it.
When we have two match patterns, e.g., because the repository layout
changed and we look for old and new style files, we can have the case
that one pattern suggests a descend into a subdir but the other pattern
would be a direct match. This was missed and only the descent was done.
Yet the other way round also has to be covered: We can't just use the
direct match because it might be that it's rejected if the type doesn't
fit (directory vs. regular file). In this case we should still descent.
Remember that we want to descend but continue checking the other
pattern first to maybe get a direct match. Introduce a new combined
return code YES_AND_RETRY to give the full information to the caller
which now can decide to fallback to a descent. While at it, use the
stat info for the directory check. Also add tests to ensure that the
order of patterns doesn't matter and we handle the above corner cases.
While for sysupdate it's fine to consume a large set of all possible
update payloads in a single directory this is not so handy for managing
and serving the update payloads. Since this large update folder is not
where the build output directly gets written to one has to create
copies and later possibly delete this added set of files.
Support matching for filenames in subdirectories by having a new **/
match pattern prefix which matches any number of nested subdirectories
or no subdirectory at all. For simplicity it's only allowed at the start
of a pattern and not a regular wildcard as the rest because the main
use case is to descend into subdirectories and only do the pattern
matching for the basenames. This way one can create a SHA256SUMS file
in the top folder and have it include all update payloads from the
release-specific (or arch-specific) subdirectories. Something similar
was already supported for directory sources where the match pattern can
start with a subdirectory path. Do also support this for SHA256SUMS for
parity while we are at it. Having the new wildcard makes mirroring also
easier because one does not have to follow the exact subdirectory layout
and one can filter by folder instead of by filename. It also makes it
possible to point the same transfer files with the new wildcard to
either a SHA256SUMS file that uses release-specific (or arch-specific)
subdirectories and includes all versions or to a SHA256SUMS file as
generated from mkosi that does not use subdirectories because it only
has files for a single version.
With the upcoming UAPI.16 JSON format we will also be able to encode
subdirectories and it makes sense to add this to SHA256SUMS for being
able to convert them. It also supports custom URLs for each entry which
is more powerful than the (arbitrary) subdirectory feature used here but
subdirectories have the advantage that they don't break mirroring.
This change also fixes the existing subdirectory handling bugs where
everything greater than two subdirectory levels failed to work because
rel_joined instead of de->d_name got used, symlinks were followed, and
we would continue silently on non-ENOENT errors.
Previously, when a library was dynamically loaded via a helper function
inside libsystemd-shared.so, the resulting dlopen ELF note was not
propagated to the invoking executable's ELF metadata.
This commit explicitly annotates each executable with the relevant
dlopen ELF notes for any optional dependencies it might potentially
load. This ensures that package managers and build systems can properly
discover runtime dependencies that are triggered indirectly.
This also switches all these notes to be defined via SD_ELF_NOTE_DLOPEN_ANCHORED().
Consequently, any notes added within unreachable or unused functions will be
automatically garbage-collected by the linker (--gc-sections) instead of
bloating the final binary.
E.g. unused p11-kit library dependency is now dropped from
systemd-repart.standalone binary.
Before:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
idn Support for internationalized domain names libidn2.so.0 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
p11-kit Support for PKCS11 hardware tokens libp11-kit.so.0 suggested
```
After:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
```
Patterns such as foo_@v/bar.efi are documented to
match files in versioned subdirectories,
but pattern_match() assumed that a field is
always followed by a literal when another element exists.
Handle a following slash as a delimiter too,
and request another recursion step
when the current path ends before that slash.
Fixes#42895.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
This file can be used for metadata about a component. It may also be
used to "disable" a component.
This brings components as a concept to a similar conceptual level as
features: both can be enabled/disabled, and carry metadata
We have two concepts: a list of files that should be extracted from an
intermediate build artifact to be used in a later build artifact, and for a
given artifact, a list "donors". The first list was called "extract", because
it is passed to the .extract_objects() meson function, and the second was
called "objects", because that is the meson parameter to specify pre-built
objects files (6350d2dbd9). But this naming is
confusing: we don't care about the 'extract' step, this is something internal
to the build machinery. And 'objects' is a very generic term.
Let's use 'export' for the stuff that is "exported" for other binaries to use,
and 'import' to say where to import from. Those terms are symmetrical and the
association between them should be intuitive. (If you think the terms are
actually assymetrical, there are precedents for confusing the import with the
import sources. E.g. in modern English, turkeys are called so because they were
imported from the Americas and guineafowls were imported from Africa via
Türkiye and all that foreign stuff is alike.)
Follow-up to #42422:
- Rename process_image() to context_process_image(), since it now
operates on a Context object.
- Use IN_SET() in image_type_can_sysupdate() instead of a switch.
- Name the return parameters of context_list_components() ret_xyz, per
our coding style.
- Drop a redundant "else" after a return in vl_method_check_new().
This is the first varlink method added to sysupdate. The D-Bus interface
(via sysupdated) will remain for now; the varlink interface will exist
in parallel.
This method can be called via:
```
varlinkctl call ./path/to/systemd-sysupdate \
io.systemd.SysUpdate.CheckNew \
'{"target":{"class":"host"}}'
SYSTEMD_SYSUPDATE_NO_VERIFY=1 \
varlinkctl call ./path/to/systemd-sysupdate \
io.systemd.SysUpdate.CheckNew \
'{"target":{"class":"component","name":"some-component"}}'
```
This includes some changes to run the integration tests again using the
varlink interface rather than running `systemd-sysupdate` directly, to
test the new interface.
This adds the scaffolding for being able to call sysupdate via varlink,
but it doesn’t yet define or implement any methods. Those will come in
following commits.
The existing `systemd-sysupdate.service` and `systemd-sysupdate.timer`
(which periodically ran `systemd-sysupdate update`) have been renamed
to `systemd-sysupdate-update.{service,timer}` to make way for a new
`systemd-sysupdate@.service` and `systemd-sysupdate.socket` file to
handle varlink activation.
Compatibility symlinks have been added for them.
This will be used in upcoming commits to varlinkify `systemd-sysupdate`;
it will need a way to identify targets over varlink, and the existing
way with a `Target` over D-Bus seems to work quite well.
`process_image()` has historically used `umount_and_freep` to clean up
the mounted directory locally, but callers to it have used
`umount_and_rmdir_and_freep`.
No directory is created after any of the error return paths in
`process_image()`, so it should probably be using
`umount_and_rmdir_and_freep` too.
This is another step towards varlinkifying the program, as it means the
various verb implementations are no longer relying on global state from
the command line.
As part of this, move init of the `Context` struct into a new
`context_from_cmdline()` function.
Additionally pass some context into config parsing `userdata` arguments,
as various config parsers were using `arg_root` via a sneaky `extern`.
This introduces no functional changes.
There’s no need for it to be heap allocated — there’s only ever one
instance of it, and it’s allocated for the lifetime of a `verb_*()`
function.
Simplify things a bit by making it stack allocated. This will also help
with upcoming commits where we introduce derived context structs to help
with varlinkifying sysupdate. By allowing `Context` to be stack
allocated we can include it in the derived context structs.
As part of this, rename `context_make_{offline,online}()` to
`context_load_{offline,online}()` for clarity (since they no longer init
the struct).
This introduces no functional changes.
`process_image()` is always called immediately before (almost) every
`context_make_online()` or `context_make_offline()`, and the structures
it allocates have the same lifetime as `Context`, so we might as well
factor them all together to reduce duplication.
This will also simplify the following commit, which changes heap
allocation of `Context`s, and simplify upcoming changes to factor out
`arg_*` handling.
The call in `verb_pending_or_reboot()` is safe because it already
validates that `arg_image` is `NULL`, hence `process_image()` will bail
out early.
This introduces no functional changes.
This makes it like all the other verbs and therefore easier to refactor.
At the same time, remove the separate `component` argument and instead
use the `component` set on the `Context`. This guards against bugs, as
various parts of the `Context` state depend on the component (for
example, `installdb_fd`) and overriding the component without also
overriding its dependent variables will lead to bugs.
If the FD has already been opened, return 1 as if opening was
successful, rather than returning 0 as if it gave `ENOENT`.
This fixes doing multiple installdb operations on a single `Context`.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Both vacuum loops silently return 0 on failure instead of propagating
the error code, hiding disk cleanup failures from the caller.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Define a new io.systemd.SysUpdate.Notify Varlink interface with a single
OnCompletedUpdate() method, and after sysupdate successfully installs an
update, invoke that method on every socket linked into
/run/systemd/sysupdate/notify/ via varlink_execute_directory(). This
gives other components a hook to react to applied updates (e.g. recompute
a TPM policy, link a freshly downloaded kernel, refresh extensions).
The notification carries the component name, the installed version and the
list of updated resources (transfer id + on-disk path). Subscribers are
free to ignore the parameters and just treat the call as a trigger.
Setting SYSTEMD_SYSUPDATE_FORCE_NOTIFY=1 forces the notification to be sent
even when no update was applied (in which case no resource list is included),
so follow-up work can be triggered unconditionally.
Fixes: #35988
This adds an operation equivalent to "systemd-sysupdate cleanup" after
an update completed (regardless if that update was entirely successful
or not). This ensures that any orphaned files are automatically cleaned
up, if they are not referenced by any transfer file's patterns anymore.
Follow-up for: d82e256bb9
Let's make sure we keep track of any file we drop into the system via a
database in /var/. This database is implemented based on symlinks, i.e.
reuses the fs as a simple database. Given the database most likely will
have <= 10 entries only (as we store *patterns* of installed file paths in
them, not the file paths themselves), this should be very efficient.
For implementation details see comments at top of
src/sysupdate/sysupdate-cleanup.c.
Just some refactoring.
The code is slightly updated, for example it now uses string_is_safe().
But mostly this just splits out code from one large file to a smaller
one.
Move the --reboot/--component= rejection into parse_argv() alongside the
other cross-option checks, and tighten TEST-72 to assert the specific
guard message rather than merely a non-zero exit.
The `pending` and `reboot` verbs, as well as the `--reboot` switch, compare
the newest installed version against the booted OS version (IMAGE_VERSION= from
os-release). When a component is selected via --component=, this compares the
component's version against the unrelated host OS version, which by design live
in separate version spaces. The result is arbitrary reboot decisions: depending
on the relative version strings sysupdate would either always or never reboot.
Refuse the combination with a clear error instead of silently performing a
bogus comparison. Correctly tracking a per-component booted version is left as a
future feature.
Fixes: https://github.com/systemd/systemd/issues/42330
unhexmem_full() ignores whitespace, so a 64-byte manifest digest field
can decode to fewer than 32 bytes. Reject that while parsing instead.
[ 83.883087] TEST-72-SYSUPDATE.sh[5995]: systemd-sysupdate: ../src/src/sysupdate/sysupdate-resource.c:581: resource_load_from_web: Assertion `h.iov_len == sizeof(instance->metadata.sha256sum)' failed.
Follow-up for 43cc7a3ef4
Assisted-by: kres (claude-opus-4-7)
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
Currently almost all the dlopens happen in libbasic or libshared code,
so the ELF dlopen notes all end up in libsystemd-shared. Many
distributions split this library and various binaries in separate
packages, and the library ends up with soft-dependencies, even though
many binaries are either completely useless or do not work at all with
the dlopen dependency. This also makes it impossible to know which
executable uses which dlopen dependency without inspecting the source
code.
If someone only wants to add the soft dependencies from libshared they
can just avoid parsing the executable binaries. By design the code in
libbasic/libshared still does the stamping too, at lower priorities, so
that libsystemd-shared will always list all the optional dependencies,
and if one wants to build a minimal system by default, they can just
parse libsystemd-shared dlopen notes, and ignore the individual
executables. But for many distribution the current setup is insufficient
and requires adding a ton of manual library dependencies, as many
executables become effectively broken or useless without the dlopen
dependencies installed (eg: resolved fails to start without libssl,
repart can do basically nothing without blkid, etc).
Add a new set of DLOPEN_<LIB> macros that wrap the dlopen_lib and also
pull in the ELF note voodoo, so that the callers get their ELF binaries
stamped too. Convert a bunch of callers to use the macro, and use
`required` dependencies for the callers that do not work without the
dlopen library being available.
The one caveat is that, in order to avoid duplicating the exact same
note in a binary due to multiple call sites, some `asm` voodoo is done
instead of the previous bare-C section-creating macro. The drawback of
this approach is that if `--gc-sections` is used to link the binary (as
we do), then binutils >= 2.36 is required for the `SHF_GNU_RETAIN` flag.
This effectively cuts off CentOS 9, so what I did here is adding an
override in meson to detect missing support, and drop `SHF_GNU_RETAIN`.
The build works, but on CentOS 9 there's no dlopen ELF notes anymore.
Given it's just that version, and it goes EOL next year, that seems ok
to me. The alternative is to drop the usage of `--gc-sections` on CentOS
9, or to accept duplicated notes everywhere, and both seem worse.
End result:
```
$ readelf --notes build/systemd-executor
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: x86 ISA needed: x86-64-baseline
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 8a0c3db54adb79ae54e1432255011aa4ab583742
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.dlopen
Owner Data size Description
FDO 0x0000006b FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"pam","description":"Support for LinuxPAM","priority":"recommended","soname":["libpam.so.0"]}]
FDO 0x0000007c FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"seccomp","description":"Support for Seccomp Sandboxes","priority":"recommended","soname":["libseccomp.so.2"]}]
FDO 0x00000090 FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"bpf","description":"Support firewalling and sandboxing with BPF","priority":"recommended","soname":["libbpf.so.1","libbpf.so.0"]}]
FDO 0x000000a0 FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"cryptsetup","description":"Support for disk encryption, integrity, and authentication","priority":"recommended","soname":["libcryptsetup.so.12"]}]
FDO 0x00000078 FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"mount","description":"Support for mount enumeration","priority":"recommended","soname":["libmount.so.1"]}]
$ readelf --notes build/systemd
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: x86 ISA needed: x86-64-baseline
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: dcd4568842e32da3e71be27db3def51c6b459994
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.dlopen
Owner Data size Description
FDO 0x00000075 FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"mount","description":"Support for mount enumeration","priority":"required","soname":["libmount.so.1"]}]
FDO 0x00000072 FDO_DLOPEN_METADATA
Dlopen Metadata: [{"feature":"selinux","description":"Support for SELinux","priority":"recommended","soname":["libselinux.so.1"]}]
```
Switch the executable-owned call sites from the plain dlopen_<lib>()
helpers to the new DLOPEN_<LIB>() macros, so that each binary that loads
an optional library at runtime now carries its own .note.dlopen note
instead of relying solely on libsystemd-shared. Each call site passes a
priority reflecting whether it hard-depends on the library (required) or
degrades gracefully without it (recommended).
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
`sysupdate --json=short components` lists the components known to
sysupdate; these are the components which something like `updatectl`
will try to update.
The `default` component represents the host, and is meant to be listed
if transfer definitions exist in (for example) `/etc/sysupdate.d`
corresponding to the host OS. This then corresponds to `TARGET_HOST` in
`updatectl` and causes it to try updating that target.
The logic for working out whether the `default` component was present
essentially boiled down to “does `{/run,/etc,/usr/lib}/sysupdate.d`
exist”, and it didn’t check whether a `.transfer` or `.conf` file
actually existed in the config directory.
This is quite the corner case, but becomes more evident on systems where
sysupdate is being used to update a portable service but not the main
OS. At that point, if `/etc/sysupdate.d` exists empty (for some reason),
`updatectl` falls over because it starts trying to update the host OS
without any configuration to do so.
So, modify `sysupdate` to more fully load the available configuration
when listing components, and query it a bit more deeply to check whether
a default component exists.
If `sysupdate` is called with various command line arguments to affect
how its configuration is loaded, do *not* say that a default component
exists, as these arguments essentially anull the possibility of a
default being used in that process.
Add an integration test based on the reproducer provided by the issue
reporter. This test has been tested to fail if the changes to
`sysupdate.c` aren’t applied — if so, the second call to `sysupdate
components` would return
`{"default":true,"components":["some-component"]}`.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Fixes: https://github.com/systemd/systemd/issues/41501
`sysupdate --json=short components` lists the components known to
sysupdate; these are the components which something like `updatectl`
will try to update.
The `default` component represents the host, and is meant to be listed
if transfer definitions exist in (for example) `/etc/sysupdate.d`
corresponding to the host OS. This then corresponds to `TARGET_HOST` in
`updatectl` and causes it to try updating that target.
The logic for working out whether the `default` component was present
essentially boiled down to “does `{/run,/etc,/usr/lib}/sysupdate.d`
exist”, and it didn’t check whether a `.transfer` or `.conf` file
actually existed in the config directory.
This is quite the corner case, but becomes more evident on systems where
sysupdate is being used to update a portable service but not the main
OS. At that point, if `/etc/sysupdate.d` exists empty (for some reason),
`updatectl` falls over because it starts trying to update the host OS
without any configuration to do so.
So, modify `sysupdate` to more fully load the available configuration
when listing components, and query it a bit more deeply to check whether
a default component exists.
If `sysupdate` is called with various command line arguments to affect
how its configuration is loaded, do *not* say that a default component
exists, as these arguments essentially anull the possibility of a
default being used in that process.
Add an integration test based on the reproducer provided by the issue
reporter. This test has been tested to fail if the changes to
`sysupdate.c` aren’t applied — if so, the second call to `sysupdate
components` would return
`{"default":true,"components":["some-component"]}`.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Fixes: https://github.com/systemd/systemd/issues/41501
Optionally prevent `context_read_definitions()` erroring out if zero
transfer definitions were found.
This commit makes no functional changes (the flag is always passed to
calls to `context_make_offline()` for the moment), but the new flag will be
used in the following commit.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: https://github.com/systemd/systemd/issues/41501
Lennart Poettering
Kai Lüke
Yu Watanabe
dongshengyuan
Zbigniew Jędrzejewski-Szmek
Philip Withnall
Armaan Sandhu
Luca Boccassi