When dealing with copying COW images, we have to make a tradeoff:
- Either we don't touch the NOCOW bit on the copied file COW and get
an instant copy because we're able to reflink, but we might get
reduced performance if the source file was COW as COW files and lots
of random writes don't play well together.
- Or we force NOCOW for the copied file, which means we have to do a
full copy as reflinking from COW files to NOCOW files or vice versa
is not supported.
In exec-invoke.c, we've opted for the first option. In nspawn.c and
discover-image.c, we've opted for the second option.
In nspawn, this applies to the --ephemeral option to make ephemeral
copies. In discover-image.c, this applies to cloning images into
/var/lib/machines. Both these features might be used to run many
machines of the same original image. We really don't want to force
a full copy onto users in these scenarios when they're expecting
reflink behavior, leading to them running out of disk space. Instead,
degraded performance in their machines is a much less severe issue,
which they will discover on their own if it affects them, at which
point they can make their original image NOCOW at which point they'll
get both the reflinks and better performance.
Given the above reasoning, let's switch nspawn.c and discover-image.c
to use COPY_NOCOW_AFTER as well instead of enabling NOCOW upfront and
forcing a copy if the original source image is COW.
Unless otherwise requested, if we're going to copy a nocow file, make the
target file nocow as well.
Aside from keeping the performance characteristics of the cow or nocow file
intact, reflinking also only works from cow to cow or nocow to nocow files.
Reflinking from cow to nocow or nocow to cow files does not work and can
easily lead to unexpected copies for users, so by keeping the nocow bit
intact across copies by default we also make sure reflinks always work.
Otherwise, when merging multiple directory trees, the output becomes
unreproducible as the directory timestamps will be changed to the current
time when copying identical directories from the second tree.
We introduce a new copy flag to achieve this behavior.
Let's freshly calculate "m" on each iteration and always start with the maximum
size we can. If sendfile() is used we must adhere to its limit of
SSIZE_MAX minus the current offset. Otherwise we can copy more, i.e.
SSIZE_MAX without any restrictions.
Also, if we get too close to having copied SSIZE_MAX, let's turn off
sendfile() for the rest.
According to copy_file_range (2), errno will be set to EOPNOTSUPP when
the file system does not support copy_file_range(). Since there is
already fallback logic in place here for other kinds of errors, add
-EOPNOTSUPP to the list of ignored errors.
This seems to be the only place where rm_rf_children() is called with a
possibly used fd, which is then passed through to rm_rf_children_impl().
This also fixes#29606.
(Tested on Fedora rawhide with kernel 6.5.6-300.fc39.x86_64.)
The subvolumes set is a set of source inodes similar to how the
denylist hashmap contains source inodes as keys. It indicates
directories in the source tree that should become subvolumes in
the target tree.
This change makes sure a data copy using copy_bytes() does not exceed the
max_bytes value when using COPY_HOLES and max_bytes stops before the next
data section.
When making ephemeral copies of files/directories whose cleanup
depends on whether they're locked or not, it's necessary to have the
lock from the very beginning, so let's support that with a new
COPY_LOCK_BSD flag.
This changes a boolean param into a proper bitflag field.
Given this only defines a single flag for now this doesn't look like
much of an improvement. But we'll add another flag shortly, where it
starts to make more sense.
The file might grow more than the amount of data we end up writing,
so let's add a flag to truncate after writing to make sure the file
is only as large as it needs to be.
The commit b640e274a7 introduced reflink()
and reflink_full(). We usually name function xyz_full() for fully
parameterized version of xyz(), and xyz() is typically a inline alias of
xyz_full(). But in this case, reflink() and reflink_full() call
different ioctl().
Moreover, reflink_full() does partial reflink, while reflink() does full
file reflink. That's super confusing.
Let's rename reflink_full() to reflink_range(), the new name is
consistent with ioctl name, and should be fine.
The kernel has had filesystem independent reflink ioctls for a
while now, let's try to use them and fall back to the btrfs specific
ones if they're not supported.
In some cases, we want to exclude a directory's contents but not
the directory itself. In other cases, we want to exclude a directory
and its contents. Let's extend the denylist logic in copy.h to support
both by changing the denylist from a set to hashmap so we can store the
deny type as the value.
We also modify the repart ExcludeFiles= option to make use of this. If
a directory to exclude ends with a "/", we'll only exclude its contents.
Otherwise, we'll exclude the full directory.
-1 was used everywhere, but -EBADF or -EBADFD started being used in various
places. Let's make things consistent in the new style.
Note that there are two candidates:
EBADF 9 Bad file descriptor
EBADFD 77 File descriptor in bad state
Since we're initializating the fd, we're just assigning a value that means
"no fd yet", so it's just a bad file descriptor, and the first errno fits
better. If instead we had a valid file descriptor that became invalid because
of some operation or state change, the other errno would fit better.
In some places, initialization is dropped if unnecessary.
RUN_WITH_UMASK was initially conceived for spawning externals progs with the
umask set. But nowadays we use it various syscalls and stuff that doesn't "run"
anything, so the "RUN_" prefix has outlived its usefulness.
When copying between filesystems, sometimes the target filesystem
might not support symlinks/fifos/sockets/... and we want to log and
ignore any failures to copy such files when copying. Let's introduce
a new flag to enable this behavior.
- Add a test that asserts that copy_tree on an existing file will fail without COPY_REPLACE
- Add a test that asserts that copy_tree with COPY_MERGE and COPY_REPLACE on an existing directory will overwrite files that already exist.
- Refactor: Move HardlinkContext to header file
- Refactor: Create `fd_copy_tree_generic` which isolates the functionality to check stat type and appropriately copy.
- Refactor: Create `fd_copy_leaf` which handles copying leaf nodes of a file tree.
This is a follow-up for f470cb6d13 which in
turn is a follow-up for a068aceafb.
The latter started to honour hidden files when deciding whether a
directory is empty. The former reverted to the old behaviour to fix
issue #23220.
It introduced a bug though: when a directory contains a larger number of
hidden entries the getdents64() buffer will not suffice to read them,
since we just allocate three entries for it (which is definitely enough
if we just ignore the . + .. entries, but not ig we ignore more).
I think it's a bit confusing that dir_is_empty() can return true even if
rmdir() on the dir would return ENOTEMPTY. Hence, let's rework the
function to make it optional whether hidden files are ignored or not.
After all, I looking at the users of this function I am pretty sure in
more cases we want to honour hidden files.
Mike Yuan
Lennart Poettering
Yu Watanabe
Daan De Meyer
Nick Rosbrook
Zbigniew Jędrzejewski-Szmek
Michael A Cassaniti
Khem Raj
Alexander Wilson