The erofs branch in make_filesystem() never passed the sector size to
mkfs.erofs, so FileSystemSectorSize= was silently ignored for erofs
partitions. mkfs.erofs defaults to the page size (e.g. 16K on aarch64),
which is too large for some use cases such as UFS with 4K blocks.
Pass -b <sector_size> to mkfs.erofs, but only when it is smaller than the
page size, since mkfs.erofs defaults to the page size and rejects larger
values. This matches how the other supported filesystems (xfs, ext4,
btrfs, f2fs, vfat) already handle sector_size.
Fixes#43238
In various cases we'll touch the same directories multiple times with
mtools: for example the /loader/ dir itself. Unfortunately mtools' mmd
does not implement a graceful "-p" switch like mkdir, but will do some
interactive name clash thing instead. We can turn this off via "-Ds
-DS", so let's do that. But that's not enough since the tool will still
fail with a non-explanatory exit status of 1. This hence ignores that
failure and proceeds anyway, under the assumption that failures to
create a directory will sooner or later be detected anyway once the
directory is to be populated and turns out not be existing because the
creation failed.
This makes some integration test/mkosi invocations work
non-interactively again.
Follow-up for f191ca982c
Two fixes to get this byte-stable:
- `fd_copy_directory()` was using `FOREACH_DIRENT_ALL`, which doesn't
give stable ordering. Read all paths, sort, then iterate.
- `mcopy -s` depends on `readdir()` ordering and thus isn't
reproducible. Implement the recursion/sorting here and only invoke
mcopy/mmd per dir.
First change increases memory usage, as we don't stream the paths
anymore, second increases the number of context switches when invoking
external tools. Both should be fine given the ESP content should usually
be pretty limited.
I'd like to write a test for this, but didn't come up with a way that
doesn't require privileges and would surface the error reliably.
do_mcopy() invoked 'mcopy -s' with the top-level sources sorted by name,
but then relied on mtools' built-in recursion to walk into subdirectories.
mtools uses platform readdir(), so the order in which it inserts entries
into the freshly-formatted FAT image inherits whatever order the source
filesystem returns. With an ext4 staging directory the directory hash
changes the readdir order between hosts (and between runs after edits),
and because FAT stores directory entries in insertion order the resulting
ESP differs byte-for-byte across otherwise identical builds.
Stop using 'mcopy -s'. Walk the source ourselves with
readdir_all_at(RECURSE_DIR_SORT) at every level and emit per-item mtools
commands interleaved per parent: each subdirectory is created with 'mmd'
before the recursion descends into it, and consecutive sibling files are
batched into a single 'mcopy' invocation so the fork cost stays
proportional to the number of directory transitions rather than to the
total number of files.
Combined with the existing 'mkfs.fat --invariant' and SOURCE_DATE_EPOCH
plumbing, this makes vfat populated via repart byte-stable regardless of
the staging filesystem's readdir() behaviour.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
I was running repart in a VM, and if failed because mkfs.vfat was
not available. But if fails quite late in the process, possibly wasting
quite a bit of work. So add a check that catches some obvious cases
where repart would fail.
The condition of whether we have the root directory is complex,
determined in part by partition_target_prepare(). I didn't think it
was worth it to recreate the full logic in the check, so in some cases
it'll not miss cases. But that's still better than having no check ;)
The updated parsing_hacks.h file uncovered a bunch of extra
things that the check-pointer-deref coccinelle script flags.
This commit fixes them to make the tree check-pointer-deref clean.
xfsprogs 6.17.0 added support for populating xfs filesystems from
directories. As this supports extended attributes unlike our current
hack with protofiles. Let's make use of the new feature in mkfs-util.c
As there's no clean way to do feature detection on the mkfs.xfs binary,
we drop support for the old hack with protofiles that we had before.
When creating ext2/ext3/ext4 filesystems, mke2fs generates a random
hash_seed for htree directory indexing. This causes non-reproducible
images even when SOURCE_DATE_EPOCH is set and the same filesystem UUID
is used.
Set the hash_seed explicitly to match the filesystem UUID, ensuring
that repeated builds with the same seed produce bit-for-bit identical
images.
Also add a test case in TEST-58-REPART to verify ext4 reproducibility
by creating the same partition twice and comparing the results.
See https://vdwaa.nl/mkosi-reproducible-arch-images.html
I used claude ai to help me with this change.
mkfs.btrfs requires that the --compress option be used together with
--rootdir, as compression only makes sense in that context (because
compression is not a persistent setting).
Right now, If --compress is specified without --rootdir, mkfs.btrfs
fails with:
ERROR: --compression must be used with --rootdir
This can occur when repart is configured with Compression= but the
partition populate logic doesn't use the --rootdir code path (eg. when
using loop device mounting to copy files after mkfs).
Add a defensive check to skip compression and emit a user-friendly
warning when compression is requested but no root directory is
provided. The warning message references the repart directive names
(Compression= and CopyFiles=) rather than low-level mkfs options to
help users understand the requirement.
This prevents crashes but doesn't enable compression, that requires
ensuring the --rootdir code path is used, which it currently is not and
will be addressed in the next patch.
Fixes: https://github.com/systemd/systemd/issues/39584
Add an fsverity flag to MkfsFlags and use it to pass the `-O verity`
option when creating an ext4 or f2fs filesystem: they share the same
argument for this.
The only other filesystem that currently supports fs-verity is btrfs and
it doesn't require a flag to be enabled when creating the filesystem.
Nothing uses this yet.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
Add a new `MakeFilesystemFlags` enum and use it to replace the existing
`quiet` and `discard` booleans on `make_filesystem()`.
Update the callers. While we're doing that, consolidate some duplicated
logic in systemd-repart.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
Change 'flags' to 'fork_flags' in preparation for using 'flags' as an
argument in the next commit.
Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
Now that mkfs.btrfs is adding support for compressing the generated
filesystem (https://github.com/kdave/btrfs-progs/pull/882), let's
add general support for specifying the compression algorithm and
compression level to use.
We opt to not parse the specified compression algorithm and instead
pass it on as is to the mkfs tool. This has a few benefits:
- We support every compression algorithm supported by every tool
automatically.
- Users don't need to modify systemd-repart if a mkfs tool learns a
new compression algorithm in the future
- We don't need to maintain a bunch of tables for filesystem to map
from our generic compression algorithm enum to the filesystem specific
names.
We don't add support for btrfs just yet until the corresponding PR
in btrfs-progs is merged.
btrfs used to default the sector size to the page size and didn't
support anything else. Since 6.7, it defaults to 4K and using 4K
makes the filesystem compatible with all page sizes. So let's make
sure we use minimum 4K as well (lower causes failures on systems with
a 4K page size) but still allow larger sector sizes if specified by
the user.
Sometimes it makes sense to hard kill a client if we die. Let's hence
add a third FORK_DEATHSIG flag for this purpose: FORK_DEATHSIG_SIGKILL.
To make things less confusing this also renames FORK_DEATHSIG to
FORK_DEATHSIG_SIGTERM to make clear it sends SIGTERM. We already had
FORK_DEATHSIG_SIGINT, hence this makes things nicely symmetric.
A bunch of users are switched over for FORK_DEATHSIG_SIGKILL where we
know it's safe to abort things abruptly. This should make some kernel
cases more robust, since we cannot get confused by signal masks or such.
While we are at it, also fix a bunch of bugs where we didn't take
FORK_DEATHSIG_SIGINT into account in safe_fork()
mcopy will set the modification time of created directories to the mtime
of the source directories but converts it to the timezone of the host.
This behavior is identical to Windows / DOS:
> The FAT file system stores time values based on the local time of the computer.
-- https://learn.microsoft.com/en-us/windows/win32/sysinfo/file-times
To achieve reproducible builds, mcopy should be invoked with TZ=UTC.
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
We only need a separate mount namespace if we're operating on a
btrfs block device so let's make sure we only unshare the mount
namespace if that's the case.
Replaces #29214
We only care about the logical sector size and if the physical sector
size isn't set and we're operating on a file, mke2fs will default the
physical sector size to the logical block size anyway.
This change makes sure that if we're operating on a block device and
set an explicit logical sector size, that doesn't affect the physical
sector size.
mkfs.btrfs refuses to operate on a block device with mounted
partitions, even if doing so is perfectly safe. An example when
this happens is when using systemd-repart with it's --image switch
to add a root partition to a /usr only image. As a workaround until
the issue is fixed, let's hide the information on mounted filesystems
from mkfs.btrfs so it doesn't fail and formats the new filesystem as
expected.
We default to quiet operation everywhere except for repart, where
we disable quiet and have the mkfs tools write to stdout.
We also make sure --quiet or equivalent is implemented for all mkfs
tools.
If no usage type is explicitly specified, ext will choose one based
on the filesystem size. Let's override this and always use the
"default" usage type so that we can create filesystems that are
initially small but might grow later without opting in to the "small"
usage type.
Yu Watanabe
Liu Zheng
Lennart Poettering
Paul Meyer
Zbigniew Jędrzejewski-Szmek
Michael Vogt
DaanDeMeyer
ZauberNerd
Chris Down
Allison Karlitskaya
Lennart Poettering
Malte PollandPaul Meyer
Malte PollandPaul Meyer