The fix in 8f1751a111 made me wonder if we could automatically detect
when pointers are accessed but when this might not be safe. Systemd
is already using a lot of `assert(dst)` and this change now forces
us to use them.
So this commit (ab)uses coccinelle to flag any pointer parameter
dereference not preceded by assert(param), ASSERT_PTR(param), or an
explicit NULL check. It adds integration into meson as a new "coccinelle"
test suite (just like clang-tidy) and is run in CI. The check is not
perfect but seems a reasonable heuristic.
For this RFC commit it is scoped to a subset, it excludes 25 dirs right
now and includes around 100. About 300 warnings left. Busywork that I am
happy to do if there is agreement that it is worth it.
With this in place we would have caught the bug from 8f1751a111 in CI:
```
FAIL: check-pointer-deref.cocci found issues in systemd/src/boot:
diff -u -p systemd/src/boot/measure.c /tmp/nothing/measure.c
--- systemd/src/boot/measure.c
+++ /tmp/nothing/measure.c
@@ -312,7 +312,6 @@ EFI_STATUS tpm_log_tagged_event(
if (err != EFI_SUCCESS)
return err;
- *ret_measured = true;
return EFI_SUCCESS;
}
```
This also adds a new POINTER_MAY_BE_NULL() for the cases when the
called function will do the NULL check (like `iovec_is_set()`).
The line to set opterr=0 was added in the initial commit in
3d090cc6f3. But afaict, this never worked as
intended, because ':' must be the first char in optstring given to
getopt_long() for it to return ':' for a missing option value. Since
this wasn't set, getopt_long() would return '?', and the missing value
would be handled as an unknown option:
$ build/systemd-journal-upload --key
Unknown option --key.
$ build/systemd-journal-upload --asdf
Unknown option --asdf.
Let's just do the standard thing:
$ build/systemd-journal-upload --key
build/systemd-journal-upload: option '--key' requires an argument
$ build/systemd-journal-upload --asdf
build/systemd-journal-upload: unrecognized option '--asdf'
In all cases, everything that we list in 'extract', we also list in
'sources'. We can simplify things by automatically appending the first
list to the second.
In the listings, move 'extract' key right below 'sources', since now
they are both "sources", just with slightly different meanings.
Let's not leak details from src/shared and src/libsystemd into
src/basic, even though you can't actually do anything useful with
just forward declarations from src/shared.
The sd-forward.h header is put in src/libsystemd/sd-common as we
don't have a directory for shared internal headers for libsystemd
yet.
Let's also rename forward.h to basic-forward.h to keep things
self-explanatory.
Fix regression introduced in a7bfb9f76b,
where busy loop can be started with a request for following logs with a
range header set with num_skip value pointing beyond the end of the
journal. In that case the reader callback returns 0 and is called
immediately again, usually causing an endless loop that is not recovered
even when new journal events are added.
The bug does not occur if num_skip is not set - in that case if no
journal entries matching the filters are added, the tight loop is
avoided by the sd_journal_wait().
To fix the issue, when no matching journal events are available, set a
flag and reuse the backoff mechanism using the sd_journal_wait().
Link: https://github.com/home-assistant/operating-system/issues/4190
We use symbols provided by unistd.h without including it. E.g.
open(), close(), read(), write(), access(), symlink(), unlink(), rmdir(),
fsync(), syncfs(), lseek(), ftruncate(), fchown(), dup2(), pipe2(),
getuid(), getgid(), gettid(), getppid(), pipe2(), execv(), _exit(),
environ, STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, F_OK, and their
friends and variants, so on.
Currently, unistd.h is indirectly included mainly in the following two paths:
- through missing_syscall.h, which is planned to covert to .c file.
- through signal.h -> bits/sigstksz.h, which is new since glibc-2.34.
Note, signal.h is included by sd-eevent.h. So, many source files
indirectly include unistd.h if newer glibc is used.
Currently, our baseline on glibc is 2.31. We need to support glibc older
than 2.34, but unfortunately, we do not have any CI environments with
such old glibc. CIFuzz uses glibc-2.31, but it builds only fuzzers, and
many files are even not compiled.
[1] says:
> Since 0.60.0 the name argument is optional and defaults to the basename of
> the first output
We specify >= 0.62 as the supported version, so drop the duplicate name in all cases
where it is the same as outputs[0], i.e. almost all cases.
[1] https://mesonbuild.com/Reference-manual_functions.html#custom_target
When `num_skip` is supplied to the `Range` header, journal-gatewayd
always returns the very last record even though it should have been
skipped. This is because the `sd_journal_next_skip` always returns
non-zero value on the first call, leading to one iteration of the
`request_reader_entries` returning the last record.
To avoid this unexpected behavior, check that the number of lines we
have skipped by is not lower than the requested skip value. If it is,
then it means there are lines which should not be returned now -
decrement the n_skip counter then and return from the function, closing
the stream if follow flag is not set.
Fixes#37954
Since 435c372ce5 added in v256,
num_entries part of the Range header is mandatory and error is returned
when it's not filled in. This makes using the "follow" argument clumsy,
because for an indefinite following of the logs, arbitrary high number
must be specified. This change makes it possible to omit it again and
documents this behavior in the man page.
Moreover, as the cursor part of the header was never mandatory, enclose
it in square brackets in the documentation as well and elaborate how
indexing works.
Following are some concrete examples of the Range header which are now
accepted:
entries= (or entries=:)
- everything starting from the first event
entries=cursor
- everything starting from `cursor`
entries=:-9:10
- last 10 events and close the connection
If the follow flag is set:
entries=:-4:10
- last 5 events, wait for 5 new and close connection
entries=:-9:
- last 10 events and keep streaming
Note that only the very last one is changing current behavior, but
reintroduces pre-v256 compatibility.
Fixes#37172
Add endpoint for listing boots. Output format mimics `journalctl
--list-boots -o json`, so it's a plain array containing index, boot ID
and timestamps of the first and last entry. Initial implementation
returns boots ordered starting with the current one and doesn't allow
any filtering (i.e. equivalent of --lines argument).
Fixes: #37573
oss-fuzz builds with --auto-features=disabled, yet we have to make
sure all fuzzers are still built when --auto-features=disabled, so
let's always build systemd-journal-remote even if it is disabled so
that we can use its objects to build fuzz-journal-remote. Instead,
when remote=disabled, we make sure we don't installed
systemd-journal-remote.
Also, let's deflatten the lists of sources in preparation for the
next commit at the same time.
In systemctl, we split out systemctl-main.c to make sure the definition
of main() is in a separate object which allows us to extract the systemctl.c
object and link it in the fuzzer target without getting a multiple definition
error when linking.
With the more efficient sync semantics it's more likely that
journal-upload-journal will try to read a partially written message.
Previously we'd fail then. Let's instead treat this gracefully,
expecting that this is either the end or will be fixed shortly (and
we'll get notified via inotify about it and recheck).
The MHD context owns the fd we watch via our event source, hence when we
destroy the context before the event source the event source might still
reference the fd that is now invalid. Hence swap the order.
Currently, when we want to add unit tests for code that is compiled into
an executable, we either compile the code at least twice (once for the
executable, and once for each test that uses it) or we create a static
library which is then used by both the executable and all the tests.
Both of these options are not ideal, compiling source files more than
once slows down the build for no reason and creating the intermediate
static libraries takes a lot of boilerplate.
Instead, let's use the extract_objects() method that meson exposes on
build targets. This allows us to extract the objects corresponding to
specific source files and use them in other executables. Because we
define all executables upfront into a dictionary, we integrate this into
the dictionary approach by adding two new fields:
- 'extract' takes a list of files for which objects should be extracted.
The extracted objects are stored in a dict keyed by the executable name
from which they were extracted.
- 'objects' takes the name of an executable from which the extracted
objects should be added to the current executable.
One side effect of this approach is that we can't build test executables
anymore without building the main executable, so we stop building test
executables unless we're also building the main executable. This allows
us to switch to using subdir_done() in all of these subdirectories to skip
parsing them if the corresponding component is disabled.
These changes get me down from 2439 => 2403 ninja targets on a full rebuild
from scratch.
Let's move some more implementation logic into functions. We keep
the logic that requires the macro in the macro and move the rest into
functions.
While we're at it, let's also make the parameter declarations of
all the string table macros less clausthrophobic.
Currently, NOTIFY_READY from daemon-util.h conflicts with NOTIFY_READY
from NotifyState from service.h so let's rename the constants to avoid
the conflict.
Now that the necessary functions from log.h have been moved to macro.h,
we can stop including log.h in macro.h. This requires modifying source
files all over the tree to include log.h instead.
In one of the next commits we'd like to introduce a concept of
optionally hashing the hostname from the machine ID. For that we we need
to optionally back gethostname_full() by code involving sd-id128, hence
let's move it from src/basic/ to src/shared/, since only there we are
allowed to use our public APIs.
- make it return NULL,
- suppress log message about reference counter of writer unless trace
logging is enabled,
- introduce cleanup function for RemoteSource.
This makes,
- When a wildcard value '*' is specified, use the first supported
compression algorithm,
- disable compression if Accept-Encoding header is unspecified or
no supported compression algorithm specified in the header,
- ignore all errors in parsing the header.
Follow-ups for cfaf78001c.
- use OrderedHashmap to manage configured compression algorithms, then
drop CompressionArgs,
- rename CompressionOpts -> CompressionConfig,
- refuse 'none' in Compression= setting, but accept boolean false, which
disables compression,
- when Compression= option is unspecified, enable all supported compression
algorithms by default,
- do not set 'none' to the Accept-Encoding header.
Michael Vogt
Lennart Poettering
Zbigniew Jędrzejewski-Szmek
Lennart Poettering
Daan De Meyer
Yu Watanabe
Jan Čermák
Jan Čermák
Andrii Chubatiuk