At various places we filter directory enumerations by inode type. Let's
add explicit support for that, so that the "struct dirent" array we
return already suppresses them.
This shortens code and makes things more robust.
Coverity flags MALLOC_SIZEOF_SAFE(de) - offsetof(DirectoryEntries,
buffer) as a potential underflow when MALLOC_SIZEOF_SAFE returns 0.
After a successful malloc the return value is at least as large as
the requested size, but Coverity cannot trace this. Add an assert
to establish the lower bound.
CID#1548020
Follow-up for 6393b847f4
Add the needed assert changes to make the code clean
for the new check-pointer-deref script.
Note that for `_cmp()` functions its a bit of a question
if we want `assert()` or `POINTER_MAY_BE_NULL()` here. The
former is more correct but it adds a (small) runtime hit. If
we use POINTER_MAY_BE_NULL() it is compiled out but strictly
speaking it is wrong (so at least we would need a comment).
Coverity flags offsetof(DirectoryEntries, buffer) + DIRENT_SIZE_MAX * 8
as a potential overflow. All operands are compile-time constants, so add
an assert_cc() to prove this at build time.
CID#1548020
Follow-up for 6393b847f4
This introduces RECURSE_DIR_UNLINK + RECURSE_DIR_UNLINK_GRACEFUL as new
return values for recurse_dir() callbacks. If either is returned the
inode currently processed will be removed and iteration continues with
the next item.
This is useful to have in the recurse_dir() mechanism itself (rather
than implement in the callback itself), due to ordering: we want to pin
the inodes via an fd while calling the callbacks, but we have to close
it before removal of the inodes. By moving this into the recurse_dir()
infra, we can implement this easily.
This partially reverts e86a492ff0.
The function getdents64() was introduced in glibc-2.30, and our baseline
on glibc is 2.31. Hence, we can assume the function always exists.
The posix_getdents() wrapper was introduced for compatibility with musl.
However, even the latest release of musl does not provide posix_getdents()
yet. Also, even with musl, by defining _LARGEFILE64_SOURCE, we can get
getdents64() and struct dirent64. Hence, the wrapper is anyway not
necessary.
Now that the necessary functions from log.h have been moved to macro.h,
we can stop including log.h in macro.h. This requires modifying source
files all over the tree to include log.h instead.
glibc exports getdents64 syscall as is, but musl exports it as
posix_getdents(). Let's introduce a simple wrapper of posix_getdents().
Note, our baseline for glibc is 2.31. Hence, we can assume getdents64()
always defined when building with glibc.
struct statx in glibc header was introduced in glibc-2.28
(fd70af45528d59a00eb3190ef6706cb299488fcd), but at that time,
sys/stat.h conflicts with linux/stat.h. Since glibc-2.30
(5dad6ffbb2b76215cfcd38c3001778536ada8e8a), sys/stat.h includes
linux/stat.h if exists.
Since now our baseline of glibc is 2.31. Hence, we can drop workarounds
for struct statx by importing linux/stat.h from newer kernel (v6.14-rc4).
-1 was used everywhere, but -EBADF or -EBADFD started being used in various
places. Let's make things consistent in the new style.
Note that there are two candidates:
EBADF 9 Bad file descriptor
EBADFD 77 File descriptor in bad state
Since we're initializating the fd, we're just assigning a value that means
"no fd yet", so it's just a bad file descriptor, and the first errno fits
better. If instead we had a valid file descriptor that became invalid because
of some operation or state change, the other errno would fit better.
In some places, initialization is dropped if unnecessary.
So far recurse_dir() will call the callback whenever we enter a
directory, and then pass the struct dirent for that directory, and an fd
for the directory the dirent is part of (i.e. the parent of the
directory we call things for). For the top-level dir the function is
invoked for we will not call the callback however, because we have no
dirent for that, and not fd for the directory the top-level dir is part
of. Let's add a flag to call it anyway, and in that case pass a NULL
dirent and -1 as directory fd.
This is useful when we want to treat the top-level dir the same as any
dir further down.
This is done opt-in since the callback must be ablet to handle a NULL
dirent and a -1 directory fd.
One of the niceties of recurse_dir()/recurse_dir_at() is that the path
argument is decoration, it's not used for actually accessing the fs in
anyway. That's very handy in environments where chroots and relative
paths are used, as we can path in any path we like and the recursion
function will suffix with whatever it discovers but will not try to make
sense of the prefix you pass.
This works great, except that the recurse_dir_at() wrapper broke that:
it adjusted the path if NULL to "." simply for the sake of making
openat() on the top work. Let's make this adjustment more local and do
it only for the openat() itself, and otherwise pass the path through the
way we got it. This means: if a caller really wants the paths that are
concatenated to start with a "." it can just pass that. This way the
caller gets full control back of the path prefix. Win!
Note that all current users of recurse_dir_at() don't pass NULL as
second arg, hence this check is without any real effect for now. It's
preparation for future uses however.
Apparently memory sanitizer doesn't grok getdents64() properly. Let's
address that by explicitly marken memory initialized by getdents64() as
unpoisoned.
We already have a similar loop twice, let's make it easier to read via
an iteration macro.
(The new macro is a bit more careful even, as it verifies the full
dirent fits into the remaining buffer when returning it)
Let's use the underlying Linux API directly, instead of
opendir()/readdir(). This makes it possible for us to do a single memory
allocation for all directory entries in common cases, instead of one for
each entry.
Apparently glibc already has a helper for this. (Not in the man pages
for Linux, but FreeBSD does document these cryptic helpers, and its
exported by glibc. That should be good enough for us.)
libc nftw() shows its age a bit, let's replace it with a more moden
infra that is built around openat(), O_PATH, statx(). This makes the
interface less prone to races and cleans up the API a bit adding
substantially more functionality.
Lennart Poettering
Luca Boccassi
Michael Vogt
Mike Yuan
Lennart Poettering
Yu Watanabe
Daan De Meyer
Zbigniew Jędrzejewski-Szmek
Dimitri Papadopoulos