This improves error output in repart if we can't fit the defined
partitions into the disk image. With this change we'll now show not only
the disk size we need (as before), but also the current one, as well as
the largest free area on disk.
This should make it a bit easier to debug disk space issues that repart
runs into.
For some reason I first commited
a73b2ad041 and then
8f47e32a3e. But the latter makes the
former obsolete and causes us to suffix paths twice.
Let's hence revert a73b2ad041 and stick to
8f47e32a3e as the latter is the ore
generic solution of the two.
This fixes repart's, systemctl's, sysusers' and tmpfiles' specifier
expansion to honour the root dir specified with --root=. This is
relevant for specifiers such as %m, %o, … which are directly sourced
from files on disk.
This doesn't try to be overly smart: specifiers referring to runtime
concepts (i.e. boot ID, architecture, hostname) rather than files on the
medium are left as is. There's certainly a point to be made that they
should fail in case --root= is specified, but I am not entirely convinced
about that, and it's certainly something we can look into later if
there's reason to.
I wondered for a while how to hook this up best, but given that quite a
large number of specifiers resolve to data from files on disks, and most
of our tools needs this, I ultimately decided to make the root dir a
first class parameter to specifier_printf().
Replaces: #16187Fixes: #16183
This is useful for provisioning initially empty secondary A/B root file
systems. We don't want those to ever be considered for automatic
mounting, for example in "systemd-nspawn --image=", hence we should
create them with the No-Auto flag turned on. Once a file system image is
dropped into the partition the flag may be turned off by the updater
tool, so that it is considered from then on.
Thew new option for this is called NoAuto. I dislike negated options
like this, but this is taken from the naming in the spec, which in turn
inherited the name from the same flag for Microsoft Data Partitions. To
minimize confusion, let's stick to the name hence.
Creating those string dynamically at runtime is slow and unnecessary.
Let's use static strings with a bit of macro magic and the let the compiler
coalesce as much as possible.
$ size build/src/shared/libsystemd-shared-248.so{.old,}
text data bss dec hex filename
2813453 94572 4584 2912609 2c7161 build/src/shared/libsystemd-shared-248.so.old
2812309 94564 4584 2911457 2c6ce1 build/src/shared/libsystemd-shared-248.so
A nice side-effect is that the same form is used everywhere, so it's easier to
figure out all variables that are used, and where each specific variable is
used.
C.f. 2b0445262a.
Note: 'const char *foo = alloca(…);' seems OK. Our coding style document and
alloca(3) only warn against using alloca() in function invocations. Declaring
both stack variable and alloca at the same time should be fine: no matter in
which order they happen, i.e. if the pointer variable is above the contents,
or the contents are above the pointer, or even if the pointer is elided by the
compiler, everything should be fine.
The previous string was "unknown", but that's wrong, because we *do*
know what we are going to do with those partitions: we leave them
unmodified, hence say "unchanged" in the output, to be clearer.
This might be useful for CopyFiles=, to reference some subdir of $TMP in
a generic way. This allows us to use the new common
system_and_tmp_specifier_table[].
Previously, we supported only "," as separator. This adds support for
"+" and makes it the documented choice.
This is to make specifying PCRs in crypttab easier, since commas are
already used there for separating volume options, and needless escaping
sucks.
"," continues to be supported, but in order to keep things minimal not
documented.
Fixe: #19205
We recently started making more use of malloc_usable_size() and rely on
it (see the string_erase() story). Given that we don't really support
sytems where malloc_usable_size() cannot be trusted beyond statistics
anyway, let's go fully in and rework GREEDY_REALLOC() on top of it:
instead of passing around and maintaining the currenly allocated size
everywhere, let's just derive it automatically from
malloc_usable_size().
I am mostly after this for the simplicity this brings. It also brings
minor efficiency improvements I guess, but things become so much nicer
to look at if we can avoid these allocation size variables everywhere.
Note that the malloc_usable_size() man page says relying on it wasn't
"good programming practice", but I think it does this for reasons that
don't apply here: the greedy realloc logic specifically doesn't rely on
the returned extra size, beyond the fact that it is equal or larger than
what was requested.
(This commit was supposed to be a quick patch btw, but apparently we use
the greedy realloc stuff quite a bit across the codebase, so this ends
up touching *a*lot* of code.)
We usually call specifier_printf() and then check the validity of
the result. In many cases, validity checkers, e.g. path_is_valid(),
refuse too long strings. This makes specifier_printf() refuse such
long results earlier.
Moreover, unit_full_string() and description field in sysuser now
refuse results longer than LONG_LINE_MAX. config_parse() already
refuses the line longer than LONG_LINE_MAX. Hence, it should be ok
to set the same value as the maximum length of the resolved string.
It's OK to specify the root dir as target directory when copying
directories. However, in that case path_extract_filename() is going to
fail, because the root dir simply has not filename.
Let's address that by moving the call further down into the loop, when
we made sure that the target dir doesn't exist yet (the root dir always
exists, hence this check is sufficient).
Moreover, in the branch for copying regular files, also move the calls
down, and generate friendly error messages in case people try to
overwrite dirs with regular files (and the root dir is just a special
case of a dir).
Altogether this makes CopyFiles=/some/place:/ work, i.e. copying some
dir on the host into the root dir of the newly created fs. Previously
this would fail with an error about the inability to extract a filename
from "/", needlessly.
When we copy files into the freshly formatted file system, the mount
point prefix must be prepended to the *target* path, not the *source*
path. Not just in code but in the log message about it, too.
This teaches repart to look for the root block device both as the
backing for /sysroot and for /sysusr/usr.
The latter is a new addition, and starts making more sense with the next
commit. It's about supporting systems that are shipped with only a /usr/
fs, but where a root fs is allocated and formatted on first boot via
systemd-repart (or a similar tool). In this case it's useful to be able
to mount the ultimate /usr/ early on without mounting the root fs
right-away (simple because the rootfs might not exist yet, and we need
the repart data encoded in /usr/ to actually format it). Hence, instead
of requiring that we mount /sysroot/ first and /sysroot/usr/ second as
we did so far, let's rearrange things slightly:
1. We mount the /usr/ file system we discover to /sysusr/usr/
2. We mount the root file system we discover to /sysroot/
3. Once both are established we bind mount /sysusr/usr/ to /sysroot/usr/
And that' it. The first two steps can happen in either order, and we can
access /usr/ with or without a rootfs being around.
This commit implements nothing of the above. Instead, it teaches
systemd-repart to check both /sysroot/ and /sysusr/ for repart drop-ins,
and use the first of these hierarchies it finds populated. This way
systemd-repart can be spawned once /usr is mounted and it will work
correctly without root fs having to exist, or we can invoke it when the
root fs is already mounted, where it also will work correctly.
Let's make the GPT partition flags configurable when creating new
partitions. This is primarily useful for the read-only flag (which we
want to set for verity enabled partitions).
This adds two settings for this: Flags= and ReadOnly=, which strictly
speaking are redundant. The main reason to have both is that usually the
ReadOnly= setting is the one wants to control, and it' more generic.
Moreover we might later on introduce inherting of flags from CopyBlocks=
partitions, where one might want to control most flags as is except for
the RO flag and similar, hence let's keep them separate.
When using systemd-repart as an installer that replicates the install
medium on another medium it is useful to reference the root
partition/usr partition or verity data that is currently booted, in
particular in A/B scenarios where we have two copies and want to
reference the one we currently use. Let's add a CopyBlocks=auto for this
case: for a partition that uses that we'll copy a suitable partition
from the host.
CopyBlocks=auto finds the partition to copy like this: based on the
configured partition type uuid we determine the usual mount point (i.e.
for the /usr partition type we determine /usr/, and so on). We then
figure out the block device behind that path, through dm-verity and
dm-crypt if necessary. Finally, we compare the partition type uuid of
the partition found that way with the one we are supposed to fill and
only use it if it matches (the latter is primarily important on
dm-verity setups where a volume is likely backed by two partitions and
we need to find the right one).
This is particularly fun to use in conjunction with --image= (where
we'll restrict the device search onto the specify device, for security
reasons), as this allows "duplicating" an image like this:
# systemd-repart --image=source.raw --empty=create --size=auto target.raw
If the right repart data is embedded into "source.raw" this will be able
to create and initialize a partition table on target.raw that carrries
all needed partitions, and will stream the source's file systems onto it
as configured.
So far we already had the CopyFiles= option in systemd-repart drop-in
files, as a mechanism for populating freshly formatted file systems with
files and directories. This adds MakeDirectories= in similar style, and
creates simple directories as listed. The option is of course entirely
redundant, since the same can be done with CopyFiles= simply by copying
in a directory. It's kinda nice to encode the dirs to create directly in
the drop-in files however, instead of providing a directory subtree to
copy in somehere, to make the files more self-contained — since often
just creating dirs is entirely sufficient.
The main usecase for this are GPT OS images that carry only a /usr/
tree, and for which a root file system is only formatted on first boot
via repart. Without any additional CopyFiles=/MakeDirectories=
configuration these root file systems are entirely empty of course
initially. To mount in the /usr/ tree, a directory inode for /usr/ to
mount over needs to be created. systemd-nspawn will do so automatically
when booting up the image, as will the initrd during boot. However, this
requires the image to be writable – which is OK for npawn and
initrd-based boots, but there are plenty tools where read-only operation
is desirable after repart ran, before the image was booted for the first
time. Specifically, "systemd-dissect" opens the image in read-only to
inspect its contents, and this will only work of /usr/ can be properly
mounted. Moreover systemd-dissect --mount --read-only won't succeed
either if the fs is read-only.
Via MakeDirectories= we now provide a way that ensures that the image
can be mounted/inspected in a fully read-only way immediately after
systemd-repart completed. Specifically, let's consider a GPT disk image
shipping with a file usr/lib/repart.d/50-root.conf:
[Partition]
Type=root
Format=btrfs
MakeDirectories=/usr
MakeDirectories=/efi
With this in place systemd-repart will create a root partition when run,
and add /usr and /efi into it as directory inods. This ensures that the
whole image can then be mounted truly read-only anf /usr and /efi can be
overmounted by the /usr partition and the ESP.
libfdisk appears to return NULL when encountering an empty partition
label, let's handle this sanely, and treat NULL and "" for the current
label as the same, but for the new label as distinct: there NULL means
nothing is set, and "" means an actual empty label.
This is similar to the --image= switch in the other tools, like
systemd-sysusers or systemd-tmpfiles, i.e. it apply the configuration
from the image to the image.
This is particularly useful for downloading minimized GPT image, and
then extending it to the desired size via:
# systemd-repart --image=foo.image --size=5G
The test occasionally fails as the umount is not yet completed when
cryptsetup close is invoked.
Both cryptsetup and losetup have supported deferred cleanup for some
time now, so use it instead to avoid races.
++ losetup -P --show --find /tmp/test-repart.dMOfYQ8UUF/zzz
+ LOOP=/dev/loop6
+ VOLUME=test-repart-11882
+ touch /tmp/test-repart.dMOfYQ8UUF/empty-password
+ cryptsetup open --type=luks2 --key-file=/tmp/test-repart.dMOfYQ8UUF/empty*** test-repart-11882
+ mkdir /tmp/test-repart.dMOfYQ8UUF/mount
+ mount -t ext4 /dev/mapper/test-repart-11882 /tmp/test-repart.dMOfYQ8UUF/mount
+ diff -r /tmp/test-repart.dMOfYQ8UUF/mount/def /tmp/test-repart.dMOfYQ8UUF/definitions
+ umount /tmp/test-repart.dMOfYQ8UUF/mount
+ cryptsetup close test-repart-11882
Device test-repart-11882 is still in use.
+ rm -rf /tmp/test-repart.dMOfYQ8UUF
If we define a partition with CopyFiles=/efi/ this should just work.
However it previously didn't because basename() would return the
trailing slash.
Let's fix this by moving things to path_extract_{directory|filename}()
This fixes the --size= switch, i.e. where we grow a disk image: after
growing it we need to expand the partition table so that its idea of the
the medium size matches the new reality. Otherwise our disk size
calculations in the subsequent steps might still use the original
ungrown size.
(This used to work, I guess this was borked when libfdisk learnt the
concept of "minimized" partition tables)
Since f17bdf8264 the test-repart was
effectively disabled, since `/dev/loop-control` is a character special
file, whereas `-f` works only on regular files. Even though we could use
`-c` to check specifically for character special files, let's use `-e`
just in case.
This nicely covers the case when optarg is optional. The same parser can be
used when the option string passed to getopt_long() requires a parameter and
when it doesn't.
The error messages are made consistent.
Also fixes a log error c&p in --crash-reboot message.
If the cleanup function returns the appropriate type, use that to reset the
variable. For other functions (usually the foreign ones which return void), add
an explicit value to reset to.
This causes a bit of code churn, but I think it might be worth it. In a
following patch static destructors will be called from a fuzzer, and this
change allows them to be called multiple times. But I think such a change might
help with detecting unitialized code reuse too. We hit various bugs like this,
and things are more obvious when a pointer has been set to NULL.
I was worried whether this change increases text size, but it doesn't seem to:
-Dbuildtype=debug:
before "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494520 Feb 16 15:06 build/systemd*
after "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494576 Feb 16 15:10 build/systemd*
now:
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494640 Feb 16 15:15 build/systemd*
-Dbuildtype=release:
before "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:09 build-rawhide/systemd*
after "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:10 build-rawhide/systemd*
now:
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:16 build-rawhide/systemd*
I would expect that the compiler would be able to elide the setting of a
variable if the variable is never used again. And this seems to be the case:
in optimized builds there is no change in size whatsoever. And the change in
size in unoptimized build is negligible.
Something strange is happening with size of libsystemd: it's bigger in
optimized builds. Something to figure out, but unrelated to this patch.
Lennart Poettering
Yu Watanabe
Zbigniew Jędrzejewski-Szmek
Frantisek Sumsal
Luca Boccassi