Let's ensure that entry seqnums remain stable and monotonic across the
entire runtime of the system, even if local storage is turned off. Let's
do this by maintainer a counter file in /run/ which we mmap() and
wherein we maintain the counter from early-boot on till late shutdown.
This takes inspiration of the kernel-seqnum file we already maintain
like that that tracks which kmsg messages we already processed. In fact,
we reuse the same code for maintaining it.
This should allow the behaviour entry seqnums to be more predictable, in
particular when journal local storage is turned off. Previously, we'd
maintain the seqnum simply by always bumping it to the maximum of the
last written entry seqnum plus one, and the biggest seqnum so far
written to the journal file on disk. If we'd never write a file on disk,
or if no journal file was existing during the initrd→seqnum transition
we'd completely lose the current seqnum position during daemon restarts
(such as the one happening during the switch-root operation).
This also will cause a journal file rotation whenever we try to write to
a journal file with multiple sequence number IDs, so that we know that
from early boot trhough the entire runtime we'll have stable sequence
numbers that do not jump, and thus can be used to determine "lost"
messages.
-1 was used everywhere, but -EBADF or -EBADFD started being used in various
places. Let's make things consistent in the new style.
Note that there are two candidates:
EBADF 9 Bad file descriptor
EBADFD 77 File descriptor in bad state
Since we're initializating the fd, we're just assigning a value that means
"no fd yet", so it's just a bad file descriptor, and the first errno fits
better. If instead we had a valid file descriptor that became invalid because
of some operation or state change, the other errno would fit better.
In some places, initialization is dropped if unnecessary.
The name "def.h" originates from before the rule of "no needless abbreviations"
was established. Let's rename the file to clarify that it contains a collection
of various semi-related constants.
../src/journal-remote/microhttpd-util.c: In function ‘check_permissions’:
../src/journal-remote/microhttpd-util.c:301:5: error: function might be candidate for attribute ‘noreturn’ [-Werror=suggest-attribute=noreturn]
301 | int check_permissions(struct MHD_Connection *connection, int *code, char **hostname) {
| ^~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Fixes#23630.
Without the size limits, oss-fuzz creates huge samples that time out. Usually
this is because some of our code has bad algorithmic complexity. For data like
configuration samples we don't need to care about this: non-rogue configs are
rarely more than a few items, and a bit of a slowdown with a few hundred items
is acceptable. This wouldn't be OK for processing of untrusted data though.
We need to set the limit in two ways: through .options and in the code. The
first because it nicely allows libFuzzer to avoid wasting time, and the second
because fuzzers like hongfuzz and afl don't support .options.
While at it, let's fix an off-by-one (65535 is the largest offset for a
power-of-two size, but we're checking the size here).
Co-authored-by: Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl>
It seems that we try to create a new file, which fails with -ENOSPC, and we
later fail when reading a file with ENODATA. journal_file_open() will return
-ENODATA if the file is too short or if journal_file_verify_header() fails.
We'll unlink a file we newly created if we fail to initialize it immediately
after creation. I'm not sure if the file we fail to open is the one we newly
created and e.g. failed to create the arena and such, or if it's the file we
were trying to rotate away from. Either way, I think we should be OK with
with a non-fully-initialized journal file.
Failed to create rotated journal: No space left on device
Failed to write entry of 2 bytes: No space left on device
sd_journal_open_files(["/tmp/fuzz-journal-remote.vELRpI.journal"]) failed: No data available
Assertion 'IN_SET(r, -ENOMEM, -EMFILE, -ENFILE)' failed at src/journal-remote/fuzz-journal-remote.c:70, function int LLVMFuzzerTestOneInput(const uint8_t *, size_t)(). Aborting.
oss-fuzz-39238: https://oss-fuzz.com/issue/4609851129462784
This is a high-level function, and it belongs in libsystemd-shared. This way we
don't end up linking a separate copy into various binaries. It would even end
up in libsystemd, where it is not needed. (Maybe it'd be removed in some
optimization phase, but it's better to not rely on that.)
$ grep -l -r -a 'path is not absolute%s' build/
build/libnss_systemd.so.2
build/pam_systemd_home.so
build/test-dlopen
build/src/basic/libbasic.a.p/path-util.c.o
build/src/basic/libbasic.a
build/src/shared/libsystemd-shared-249.so
build/test-bus-error
build/libnss_mymachines.so.2
build/pam_systemd.so
build/libnss_resolve.so.2
build/libnss_myhostname.so.2
build/libsystemd.so.0.32.0
build/libudev.so.1.7.2
$ grep -l -r -a 'path is not absolute%s' build/
build/src/shared/libsystemd-shared-251.a.p/parse-helpers.c.o
build/src/shared/libsystemd-shared-251.a
build/src/shared/libsystemd-shared-251.so
No functional change.
GIT_VERSION is not available as a config.h variable, because it's rendered
into version.h during builds. Let's rework jinja2 rendering to also
parse version.h. No functional change, the new variable is so far unused.
I guess this will make partial rebuilds a bit slower, but it's useful
to be able to use the full version string.
The approach to use '''…'''.split() instead of a list of strings was initially
used when converting from automake because it allowed identical blocks of lines
to be used for both, making the conversion easier.
But over the years we have been using normal lists more and more, especially
when there were just a few filenames listed. This converts the rest.
No functional change.
by always calling journal_remote_server_destroy, which resets global
variables like journal_remote_server_global. It should prevent crashes like
```
Assertion 'journal_remote_server_global == NULL' failed at src/journal-remote/journal-remote.c:312, function int journal_remote_server_init(RemoteServer *, const char *, JournalWriteSplitMode, _Bool, _Bool)(). Aborting.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==24769==ERROR: AddressSanitizer: ABRT on unknown address 0x0539000060c1 (pc 0x7f23b4d5818b bp 0x7ffcbc4080c0 sp 0x7ffcbc407e70 T0)
SCARINESS: 10 (signal)
#0 0x7f23b4d5818b in raise /build/glibc-eX1tMB/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:51:1
#1 0x7f23b4d37858 in abort /build/glibc-eX1tMB/glibc-2.31/stdlib/abort.c:79:7
#2 0x7f23b5731809 in log_assert_failed systemd/src/basic/log.c:866:9
```
The meson default for static_library() are:
build_by_default=true, install=false. We never interact with the
static libraries, and we only care about them as a stepping-stone towards
the installable executables or libraries. Thus let's only build them if
they are a dependency of something else we are building.
While at it, let's drop install:false, since this appears to be the default.
This change would have fixed the issue with lib_import_common failing
to build too: we wouldn't attempt to build it.
In practice this changes very little, because we generally only declare static
libraries where there's something in the default target that will make use of
them. But it seems to be a better pattern to set build_by_default to false.
Preparatory commit; before JournalFile can stop hanging onto its
copy of MMapCache, all these users need to find another way.
Most of the time these callers already have the MMapCache onhand,
so it's no big deal for them to just supply it.
journal_file_rotate() in particular needed to change, and it
seemed wise to not use the mmap_cache_fd_cache() accessor on
f->cache_fd, instead requiring the caller supply the cache to
use. This was done with an eye towards a potential future where
the journal_file_archive() isolates the cache_fd to a private
cache, which the newly rotated-to file wouldn't be allowed to
use. It's no biggie for the existing callers to just provide the
appropriate surviving cache.
Basically the mmap_cache_fd_cache() accessor was added just for
journal-verify.c's (ab)use of the mmap-cache. Which, if the
ugly singleton MMapCache assumption ever goes away, can be
cleaned up to simply use a separate MMapCache for those search
arrays.
Currently, all the logic related to writing journal files lives in
journal-file.c which is part of libsystemd (sd-journal). Because it's
part of libsystemd, we can't depend on any code from src/shared.
To allow using code from src/shared when writing journal files, let's
gradually move the write related logic from journal-file.c to
journald-file.c in src/journal. This directory is not part of libsystemd
and as such can use code from src/shared.
We can safely remove any journal write related logic from libsystemd as
it's not used by any public APIs in libsystemd.
This commit introduces the new file along with the JournaldFile struct
which wraps an instance of JournalFile. The goal is to gradually move
more functions from journal-file.c and fields from JournalFile to
journald-file.c and JournaldFile respectively.
This commit also modifies all call sites that write journal files to
use JournaldFile instead of JournalFile. All sd-journal tests that
write journal files are moved to src/journal so they can make use of
journald-file.c.
Because the deferred closes logic is only used by journald, we move it
out of journal-file.c as well. In journal_file_open(), we would wait for
any remaining deferred closes for the file we're about to open to complete
before continuing if the file was not newly created. In journald_file_open(),
we call this logic unconditionally since it stands that if a file is newly
created, it can't have any outstanding deferred closes.
No changes in behavior are introduced aside from the earlier execution
of waiting for any deferred closes to complete when opening a new journal
file.
When using "capture : true" in custom_target()s the mode of the source
file is not preserved when the generated file is not installed and so
needs to be tweaked manually. Switch from output capture to creating the
target file and copy the permissions from the input file.
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
rearrange_stdio() invalidates specified fds even on failure, which means
we should always invalidate the fds we pass in no matter what. Let's
make this explicit by using TAKE_FD() for that everywhere.
Note that in many places we such invalidation doesnt get us much
behaviour-wise, since we don't use the variables anymore later. But
TAKE_FD() in a way is also documentation, it encodes explicitly that the
fds are invalidated here, so I think it's a good thing to always make
this explicit here.
When journald is rotating a file, we'd like to log the reason at
LOG_INFO or higher instead of LOG_DEBUG. For journalctl --header,
logging the reason at a level higher than LOG_DEBUG doesn't really
make sense. To accomodate both use cases, make the log level used
by journal_file_rotate_suggested() configurable.
Let's define two helpers strdupa_safe() + strndupa_safe() which do the
same as their non-safe counterparts, except that they abort if called
with allocations larger than ALLOCA_MAX.
This should ensure that all our alloca() based allocations are subject
to this limit.
afaics glibc offers three alloca() based APIs: alloca() itself,
strndupa() + strdupa(). With this we have now replacements for all of
them, that take the limit into account.
Lennart Poettering
Yu Watanabe
Jan Janssen
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
berenddeschouwerandBerend De Schouwer
Cristian Rodríguez
Daniel Braunwarth
David Tardon
Daan De Meyer
Evgeny Vereshchagin
Vito Caputo
Luca Boccassi
Christian Brauner