An ‘acquire’ (download) part, and an ‘install’ (apply) part.
Following commits will expose these as separate verbs and D-Bus methods,
but this commit is the one which rearranges the internals.
If doing an ‘install’, a mirror version of the ‘acquire’ has to happen
first to make sure the transfer’s internal state is correct.
‘Acquire’ can require an internet connection, but ‘install’ will always
work with `--offline` specified.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: https://github.com/systemd/systemd/issues/34814
Modify the vacuum implementation to preferentially vacuum partial or
pending transfers first (unless protected) as they are meant to be
fairly transitory, and ones which are hanging around have probably been
forgotten about and/or are out of date.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: https://github.com/systemd/systemd/issues/34814
Instead of using a random temporary path for file transfers, use a
predictable one which indicates whether the transfer is partially
complete or pending installation. Similarly for partitions.
This is another step towards being able to split the ‘update’ step into
‘acquire’ and ‘install’.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
Helps: https://github.com/systemd/systemd/issues/34814
Let's make sure we can safely load configuration files based on a root
fd, including XAT_FDROOT.
(While we are at it, always pass path to root fs before fd to root fs,
following our recently updated CODING_STYLE)
Since c5de7b14ae
file searching implies a new mount api syscall by default,
to trigger automounts.
But, this is not necessary in most cases, e.g. when chasing
syspath in sd-device (actually this causes regression in umockdev,
see https://github.com/martinpitt/umockdev/issues/271).
Another example is reading unit files, especially .network files,
as automount may trigger mounting network filesystems...
Also, when this is used in NSS plugins, programs that load the
plugins may fail because of spuriously configured seccomp. See #38565.
Let's not trigger automount by default, and do only when explicitly
requested.
This introduces CHASE_TRIGGER_AUTOFS, and use it in
- service manager,
- bootctl and finding ESP/xbootldr,
- sysupdate,
- mountfsd,
- systemd-mount.
There may be several more places we should trigger automount, but let's
do that later.
Follow-up for c5de7b14ae.
Fixes#38565.
Replaces #38569.
Co-authored-by: Luca Boccassi <luca.boccassi@gmail.com>
This reverts commit 490aa05ca1.
As commented https://github.com/systemd/systemd/pull/38569#discussion_r2284978273,
the commit makes autofs check bypassed. Before the commit, when
CHASE_NO_AUTOFS is set, we did not shortcut chasing paths, and refused
any autofs mount points in the path. However, with the commit, the flag
was swapped but even when CHASE_AUTOFS is unset, the autofs check may be
skipped.
To fix the issue, rather than swapping the flag, we should introduce
another flag, say CHASE_TRIGGER_AUTOFS. This revert the commit, and in a
later commit, the new flag will be introduced.
Since https://github.com/systemd/systemd/commit/c5de7b14ae2e08d267d8d75bc88934ac6aa7dcd6
file searching implies a new mount api syscall by default,
to trigger automounts.
This is problematic in NSS plugins, as they are dlopen'ed inside
processes by glibc, for two reasons.
First of all, potentially searching on a networked filesystem
automount could lead to nasty surprises, such as the process
responsible for setting up the network filesystem trying to
search on that same filesystem.
More importantly, the new mount api syscall was never part of
the filesystem seccomp filter that we provide by default, and
given mounting/remounting/bind mounting is one of the possible
ways to bypass sandboxing it is very likely not allowed when
custom filters are used in sandboxed processes, if they don't
need to do these operations otherwise.
The filesystem seccomp mask we provide has been updated, however
this only takes effect on the next restart of a service. When
systemd is upgraded via a package upgrade, the new nss plugin is
installed and will be immediately dlopen'ed by glibc when needed,
without waiting for the process to restart, which means the existing
seccomp filter applies, causing the filter to trigger.
Given it's not really possible for any arbitrary program to
predict which NSS modules glibc will load, given programs do not
configure that and instead nsswitch is set up by the sysadmin,
it's impossible to handle at each process level. It's also not
possible to know when it will be triggered, given the plugin
is not linked in each binary tools like need-restart cannot
even pre-emptively restart services that may be affected.
This means in practice, upgrading from systemd << v258 to >= v258
requires a reboot to avoid either subtle or catastrophic system
failures.
By avoiding to trigger automounts in nss-systemd we can avoid
both issues.
userdb drop-ins are searched for in:
/etc/userdb/
/run/userdb/
/run/host/userdb/
/usr/local/lib/userdb/
/usr/lib/userdb/
none of which are supported as automounts anyway.
Note that this happens only when the userdbd service is not running,
as otherwise nss-systemd will go through the varlink IPC, rather than
doing the searches in-process.
So invert CHASE_NO_AUTOFS to CHASE_AUTOFS and set it in the places where
we do want to trigger automounts, like looking for the ESP.
Follow-up for https://github.com/systemd/systemd/commit/c5de7b14ae2e08d267d8d75bc88934ac6aa7dcd6
Fixes https://github.com/systemd/systemd/issues/38565
We use symbols provided by unistd.h without including it. E.g.
open(), close(), read(), write(), access(), symlink(), unlink(), rmdir(),
fsync(), syncfs(), lseek(), ftruncate(), fchown(), dup2(), pipe2(),
getuid(), getgid(), gettid(), getppid(), pipe2(), execv(), _exit(),
environ, STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, F_OK, and their
friends and variants, so on.
Currently, unistd.h is indirectly included mainly in the following two paths:
- through missing_syscall.h, which is planned to covert to .c file.
- through signal.h -> bits/sigstksz.h, which is new since glibc-2.34.
Note, signal.h is included by sd-eevent.h. So, many source files
indirectly include unistd.h if newer glibc is used.
Currently, our baseline on glibc is 2.31. We need to support glibc older
than 2.34, but unfortunately, we do not have any CI environments with
such old glibc. CIFuzz uses glibc-2.31, but it builds only fuzzers, and
many files are even not compiled.
Admittedly, some of our glyphs _are_ special, e.g. "O=" for SPECIAL_GLYPH_TOUCH ;)
But we don't need this in the name. The very long names make some invocations
very wordy, e.g. special_glyph(SPECIAL_GLYPH_SLIGHTLY_UNHAPPY_SMILEY).
Also, I want to add GLYPH_SPACE, which is not special at all.
Follow-up for 7f6af95dab
- Allocate internal buf on the stack, memdup() only at the end.
This ensures we're able to handle OOM gracefully, i.e.
return -EAGAIN on OOM while still emptying socket buffer.
- Do not treat empty notify message as error.
- Raise log level since all callers log loudly anyway.
Optional features allow distros to define sets of transfers that can
be enabled or disabled by the system administrator. This is useful for
situations where a distro may want to ship some resources version-locked
to the core OS, but many people have no need for the resource, such as:
development tools/compilers, drivers for specialized hardware, language
packs, etc
We also rename sysupdate.d/*.conf -> sysupdate.d/*.transfer, because
now there are more than one type of definition in sysupdate.d/. For
backwards compat, we still load *.conf files as long as no *.transfer
files are found and the *.conf files don't try to declare themselves
as part of any features
Fixes https://github.com/systemd/systemd/issues/33343
Fixes https://github.com/systemd/systemd/issues/33344
In mkosi, I want to add a sysupdate verb to wrap systemd-sysupdate.
The definitions will be picked up from mkosi.sysupdate/ and passed
to systemd-sysupdate. I want users to be able to write transfer
definitions that are independent of the output directory used by
mkosi. To make this possible, it should be possible to specify the
directory that transfer sources should be looked up in on the sysupdate
command line. Let's allow this via a new --transfer-source= option.
Additionally, transfer sources that want to take advantage of this
feature should specify PathRelativeTo=directory to indicate the configured
Path= is interpreted relative to the tranfer source directory specified
on the CLI.
This allows for the following transfer definition to be put in
mkosi.sysupdate:
"""
[Transfer]
ProtectVersion=%A
[Source]
Type=regular-file
Path=/
PathRelativeTo=directory
MatchPattern=ParticleOS_@v.usr-%a.@u.raw
[Target]
Type=partition
Path=auto
MatchPattern=ParticleOS_@v
MatchPartitionType=usr
PartitionFlags=0
ReadOnly=1
"""
Previously, if the callout binary (i.e. sd-pull, sd-import) failed
gracefully, we'd return its exit status from the event loop and thus
from run_callout(). Of course, exit status is a positive number in the
event of failure. Which means that we completely ignore the callout
binary failing, and instead continue using whatever it managed to
download before failing.
This is bad for obvious reasons, not the least of which is installing
a half-downloaded OS. This also means that we would completely ignore
failed signature checks 😬️
A previous commit made sysupdate recognize installed versions where some
transfers are missing. This commit teaches sysupdate how to correctly
repair these incomplete versions.
Previously, if you had a incomplete installation of the OS booted, and
ran sysupdate in an attempt to repair it, sysupdate would make things
worse by creating copies of the currently-booted partitions in the
inactive slots. Then at boot you have two identical partitions, with
identical labels an UUIDs, and end up with a mess.
With this commit, sysupdate is able to recognize situations where it can
simply download the missing transfers and leave the rest of the system
undistrubed.
Partial fix for https://github.com/systemd/systemd/issues/33339
Makes it possible to specify URLs to a changelog and an appstream
catalog XML in the sysupdate.d/*.conf files. This will be passed along
to the clients of systemd-sysupdated, which can then present this data.
We set up a NOTIFY_SOCKET to get download progress notifications from
each individual import helper. Along with the number of import jobs we
have to run, this gives an overall progress value which we report using
sd_notify
The MatchPattern= in Target specification is documented as mandatory,
but if not defined sysupdate will assume the same value as definied
in Source specification and continue execution. Print this
assumptions to the user
Sometimes it makes sense to hard kill a client if we die. Let's hence
add a third FORK_DEATHSIG flag for this purpose: FORK_DEATHSIG_SIGKILL.
To make things less confusing this also renames FORK_DEATHSIG to
FORK_DEATHSIG_SIGTERM to make clear it sends SIGTERM. We already had
FORK_DEATHSIG_SIGINT, hence this makes things nicely symmetric.
A bunch of users are switched over for FORK_DEATHSIG_SIGKILL where we
know it's safe to abort things abruptly. This should make some kernel
cases more robust, since we cannot get confused by signal masks or such.
While we are at it, also fix a bunch of bugs where we didn't take
FORK_DEATHSIG_SIGINT into account in safe_fork()
`MatchPattern` for regular-file and directory as target can now match
subdirectories This is useful to install files for examples in `.extra.d`
directories:
```
[Target]
Type=regular-file
Path=/EFI/Linux
PathRelativeTo=boot
MatchPattern=gnomeos_@v.efi.extra.d/apparmor.addon.efi
```
The if the directories in the path do not exist, they will be created. Whereas
the part in `Path` is not created.
The article "a" goes before consonant sounds and "an" goes before vowel
sounds. This commit changes an to a for UKI, UDP, UTF-8, URL, UUID, U-Label, UI
and USB, since they start with the sound /ˌjuː/.
While we are at it, replace the sloppy use of filename_is_valid() by the
less sloppy filename_part_is_valid() (as added by the preceeding
commit), since we don#t want to be too restrictive here. (After all,
version strings invalid as standalone filenames might be valid as part
of filenames, and hence we should allow them).
As described in the BLS, we should place binaries into the XBOOTLDR
directory if it is available, otherwise into the ESP. Thus, we might
need to put binaries into /boot or into /efi depending on the existence
of the XBOOTLDR partition.
With this change, we introduce a new PathRelativeTo= config option that
makes this functionality possible
Chasing symlinks is a core function that's used in a lot of places
so it deservers a less verbose names so let's rename it to chase()
and chaseat().
We also slightly change the pattern used for the chaseat() helpers
so we get chase_and_openat() and similar.
Instead of exposing just the partition type UUID, let's expose the
GptPartitionType struct, which has a lot more information available
in a much more accessible way.
Also, let's get rid of SECONDARY/OTHER in PartitionDesignator. These
were only there to support preferred architectures in dissect-image.c,
but we can easily handle that by comparing architectures when we decide
whether to override a partition. This is done in a new function
compare_arch().
Philip Withnall
Lennart Poettering
Yu Watanabe
Luca Boccassi
Mike Yuan
Daan De Meyer
Zbigniew Jędrzejewski-Szmek
Matthias Lisin
Adrian Vovk
Abderrahim Kitouni
Burak Gerz
Valentin David
Joerg Behrmann