Currently, ExcludeFiles= supports excluding directories on the host
from being copied. Let's extend this to also support preventing files
from being copied into specific directories in the partition by adding
a new option ExcludeFilesTarget=. An example where this is useful is
when setting up btrfs subvolumes in the top level that are intended to
be mounted into specific locations, so /usr would be stored in @usr,
/home in @home, .... To accomplish this, we need to copy /usr to @usr
and prevent any files from being copied into /usr in the partition,
which with this commit, we'd be able to do as follows:
```
[Partition]
CopyFiles=/usr:@usr
ExcludeFilesTarget=/usr
```
There is really no reason to go out of our way to create the config files as
the unprivileged user. And in the logs, the runas calls are quite verbose,
distracting from the interesting stuff. Also add .defs/.imgs/.root to the
temporary paths to make them easier to distinguish in the logs.
Until now, we always excluded the top level directories that were
covered by child partition mount points, regardless of the source
directory and the target directory of the copy files operation.
This means that even if we were populating a XBOOTLDR partition, if
there was an EFI partition in the image, we'd exclude /boot
unconditionally, leading to the XBOOTLDR partition to be empty.
Also, because of the same cause, if we were copying a nested source
directory (e.g. /abc/def) to the root directory in the root
partition, if /abc/def/usr existed and was populated with files and
directories, the root partition would have those files under /usr,
even if a /usr partition was defined.
To fix these issues, instead of unconditionally excluding the top
level partition mount points under <source>, let's make sure that
when we're copying files from any source directory to the root
directory of a root partition, that we exclude the partition mount
point directories under the source directory instead of the top
level ones.
systemd-repart needs to find mkfs.ext4 for the test.
This is located in the directory /usr/sbin on openSUSE Tumbleweed.
But since the variable ALWAYS_SET_PATH in /etc/login.defs is set to yes,
su re-initializes the $PATH variable and removes /usr/sbin.
Hence, mkfs.ext4 is not found and the test fails.
Using setpriv instead of su fixes this issue and is more appropriate to
do the switch user task from root.
[zjs: move setpriv to $BASICTOOLS and force-push to retrigger CI]
In some cases, we want to exclude a directory's contents but not
the directory itself. In other cases, we want to exclude a directory
and its contents. Let's extend the denylist logic in copy.h to support
both by changing the denylist from a set to hashmap so we can store the
deny type as the value.
We also modify the repart ExcludeFiles= option to make use of this. If
a directory to exclude ends with a "/", we'll only exclude its contents.
Otherwise, we'll exclude the full directory.
(The one case that is left unchanged is '< <(subcommand)'.)
This way, the style with no gap was already dominant. This way, the reader
immediately knows that ' < ' is a comparison operator and ' << ' is a shift.
In a few cases, replace custom EOF replacement by just EOF. There is no point
in using someting like "_EOL" unless "EOF" appears in the text.
This reverts commit e59678b2cf.
We also modify the repart integration tests to make them pass with the
changes in this commit. In short, we have to make sure every file is
owned by the user executing repart. We use tee instead of cat since it
makes that easier. This also has the benefit of improving debugability
as seeing the config file contents on stdout makes it easier to know
which test is failing.
So, i think "erofs" is probably the better, more modern alternative to
"squashfs". Many of the benefits don't matter too much to us I guess,
but there's one thing that stands out: erofs has a UUID in the
superblock, squashfs has not. Having an UUID in the superblock matters
if the file systems are used in an overlayfs stack, as overlayfs uses
the UUIDs to robustly and persistently reference inodes on layers in
case of metadata copy-up.
Since we probably want to allow such uses in overlayfs as emplyoed by
sysext (and the future syscfg) we probably should ramp up our erofs game
early on. Hence let's natively support erofs, test it, and in fact
mention it in the docs before squashfs even.
--include-partitions and --exclude-partitions now fully exclude
partitions from repart. Whenever a partition type is excluded, we
don't take any partitions of that type into account at all when
running systemd-repart.
--skip-partitions= is introduced to do what --exclude-partitions did
previously. Any skipped partitions are taken into acount when doing
size calculations, but are not yet populated.
Why do we need both concepts? Exclusion is needed so that we can
use shared repart definitions to generate bootable and non-bootable
images. When generating a non-bootable image, we use --exclude-partitions
to exclude the ESP partition. Skipping is needed so that we can
populate the root partition while skipping the ESP partition, get
the roothash of the root partition, use that to generate a UKI, and
finally populate the ESP partition with the UKI included.
To make sure rootless mode keeps working, let's run all repart
integration tests that we can without root privileges. The only ones
we need to keep running with root privileges are the tests that operate
on a block/loop device and those that use --image=.
This is useful to force off fancy unicode glyph use (i.e. use "->"
instead of "→"), which is useful in tests where locales might be
missing, and thus control via $LC_CTYPE is not reliable.
Use this in TEST-58, to ensure the output checks we do aren't confused
by missing these glyphs being unicode or not.
This commit adds a new Verity= setting to repart definition files
with two possible values: "data" and "hash".
If Verity= is set to "data", repart works as before, and populates
the partition with the content from CopyBlocks= or CopyFiles=.
If Verity= is set to "hash", repart will try to find a matching
data partition with Verity=data and equal values for CopyBlocks=
or CopyFiles=, Format= and MakeDirectories=. If a matching data
partition is found, repart will generate verity hashes for that
data partition in the verity partition. The UUID of the data
partition is set to the first 128 bits of the verity root hash. The
UUID of the hashes partition is set to the final 128 bits of the
verity root hash.
Fixes#24559
The original workaround didn't work, as `systemd-repart` kept failing
even when the `/dev/loopX` device was present:
```
[ 13.959419] H testsuite-58.sh[280]: + LOOP=/dev/loop1
[ 13.959636] H testsuite-58.sh[280]: + :
[ 13.959764] H testsuite-58.sh[280]: + test -e /dev/loop1
[ 13.959895] H testsuite-58.sh[280]: + break
[ 13.960023] H testsuite-58.sh[280]: + systemd-repart --pretty=yes --definitions=/tmp/testsuite-58-sector/ --seed=750b6cd5c4ae4012a15e7be3c29e6a47 --empty=require --dry-run=no /dev/loop1
[ 13.970538] H testsuite-58.sh[363]: Device '/dev/loop1' has no dm-crypt/dm-verity device, no need to look for underlying block device.
[ 13.970538] H testsuite-58.sh[363]: Failed to determine canonical path for '/dev/loop1': No such file or directory
[ 13.970538] H testsuite-58.sh[363]: Failed to open file or determine backing device of /dev/loop1: No such file or directory
```
It seems there exists a short time period that we cannot see the
loopback device after `losetup` is finished:
```
testsuite-58.sh[367]: ++ losetup -b 1024 -P --show -f /tmp/testsuite-58-sector-1024.img
kernel: loop1: detected capacity change from 0 to 204800
testsuite-58.sh[285]: + LOOP=/dev/loop1
testsuite-58.sh[285]: + systemd-repart --pretty=yes --definitions=/tmp/testsuite-58-sector/ --seed=750b6cd5c4ae4012a15e7be3c29e6a47 --empty=require --dry-run=no /dev/loop1
testsuite-58.sh[368]: Device '/dev/loop1' has no dm-crypt/dm-verity device, no need to look for underlying block device.
testsuite-58.sh[368]: Failed to determine canonical path for '/dev/loop1': No such file or directory
testsuite-58.sh[368]: Failed to open file or determine backing device of /dev/loop1: No such file or directory
```
The Ubuntu CI on ppc64el seems to have a issue on tmpfs, and files
may not be fsynced. See c10caebb98.
For safety, let's use /var/tmp to store disk images.
Luca Boccassi
Zbigniew Jędrzejewski-Szmek
Daan De Meyer
Frantisek Sumsal
Thomas Blume
Franck Bui
Lennart Poettering
Yu Watanabe
Luca Boccassi