Almost all programs are simple and have only a single command. When
printing options, let's just use the first COMMAND block found by default.
This means we can drop the explicit program-name argument and reduce
copy&paste boilerplate.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The synopsis now marks the input files as
optional, since the local journal is read when none are given, and
--url= as optional, since URL= may come from journal-upload.conf,
matching the man page.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The synopsis now marks the input files as
optional, since input may also come from --url=, --getter=, --listen-*=
or passed file descriptors, matching the man page. The note about
sd_listen_fds() is kept as the footer.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The abstract is changed from the noun phrase
"HTTP server for journal events." to the imperative "Serve journal events
over HTTP.", consistent with the other commands, and the meaningless
"..." placeholder is dropped from the synopsis (the tool takes no
positional arguments).
Since 4c0d8d9673, most dlopen notes are
set at the beginning of the executables. Let's manage all dlopen notes
there, rather than setting them where dlopen is called.
Note that the only exceptions are the LIBBPF_NOTE for networkd and
nsresource. Since dlopen_bpf() is wrapped in an `#if` guard, the notes
are instead set within the corresponding functionality.
As a result, the DLOPEN_FOO() wrapper macros are no longer needed and
can be dropped completely.
MHDDaemonWrapper_free() unconditionally unrefs both event sources, but
setup_microhttpd_server() could return before initializing either pointer.
Zero-initialize the wrapper so cleanup after MHD startup failures safely
unrefs NULL.
Follow-up for 3c67c8bd4c
compression_config_put() frees the COMPRESSION_NONE value when a later
algorithm is selected, but lefts the map entry pointing at it. Remove the
entry first so subsequent parsing, mangling, and teardown cannot access or
free stale memory.
Follow-up for c259c9e253
Previously, even though sd-journal does not support bzip2 and gzip
compression, the dlopen ELF notes for those libraries were still being
attached to libsystemd.so and various journal-handling executables.
This occurred because the unified compression/decompression interfaces
handled all formats unconditionally, causing the compiler and linker to
pull in all associated dlopen notes across the board.
To resolve this, split these functions into generic and journal-specific
variants (e.g., introducing compress_blob_journal() and decompress_blob_journal()).
The journal-specific variants only handle formats actually supported by
the journal (LZ4, XZ, and ZSTD).
By updating sd-journal, journald, journalctl, and related utilities to
use these new `_journal` interfaces and switching them to the narrower
COMPRESS_JOURNAL_NOTE macro, we ensure that unnecessary dlopen notes
(for bzip2 and gzip) are no longer embedded into these binaries.
Previously, when a library was dynamically loaded via a helper function
inside libsystemd-shared.so, the resulting dlopen ELF note was not
propagated to the invoking executable's ELF metadata.
This commit explicitly annotates each executable with the relevant
dlopen ELF notes for any optional dependencies it might potentially
load. This ensures that package managers and build systems can properly
discover runtime dependencies that are triggered indirectly.
This also switches all these notes to be defined via SD_ELF_NOTE_DLOPEN_ANCHORED().
Consequently, any notes added within unreachable or unused functions will be
automatically garbage-collected by the linker (--gc-sections) instead of
bloating the final binary.
E.g. unused p11-kit library dependency is now dropped from
systemd-repart.standalone binary.
Before:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
idn Support for internationalized domain names libidn2.so.0 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
p11-kit Support for PKCS11 hardware tokens libp11-kit.so.0 suggested
```
After:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
```
Extend the 'install' keyword for executable definitions to accept four
modes—'yes', 'no', 'both', and 'static'—to elegantly manage the creation
and installation of both shared and statically-linked (.standalone) binaries.
- 'yes' / 'no': Standard behavior (mapped to true/false).
- 'both': Installs both the shared and static variants.
- 'static': Installs the static variant under the original name, while
suffixing the uninstalled shared variant with '.shared'.
With this change, any arbitrary executable can now have its `.standalone`
variant built on demand simply by invoking `ninja <target>.standalone`.
For example, `varlinkctl` did not previously support a standalone variant,
but it can now be built explicitly via `ninja varlinkctl.standalone`.
These `.standalone` binaries are not built by default unless explicitly
specified as a ninja target or enabled via `-Dstandalone-binaries=true`.
Thus, the default build time should remain unaffected.
This centralisation eliminates a massive amount of boilerplate and duplicated
target declarations across almost all subdirectories (e.g., systemd-repart,
systemd-tmpfiles, systemd-shutdown, and systemd-report tools).
This handling with tmpfiles was dropped in
29444df23b. So let's stop the build system
to create that directory. Actually it is create by the service just
fine, including correct access mode.
We have two concepts: a list of files that should be extracted from an
intermediate build artifact to be used in a later build artifact, and for a
given artifact, a list "donors". The first list was called "extract", because
it is passed to the .extract_objects() meson function, and the second was
called "objects", because that is the meson parameter to specify pre-built
objects files (6350d2dbd9). But this naming is
confusing: we don't care about the 'extract' step, this is something internal
to the build machinery. And 'objects' is a very generic term.
Let's use 'export' for the stuff that is "exported" for other binaries to use,
and 'import' to say where to import from. Those terms are symmetrical and the
association between them should be intuitive. (If you think the terms are
actually assymetrical, there are precedents for confusing the import with the
import sources. E.g. in modern English, turkeys are called so because they were
imported from the Americas and guineafowls were imported from Africa via
Türkiye and all that foreign stuff is alike.)
This makes each `foo_cflags` dependency define a `SYSTEMD_CFLAGS_MARKER_FOO`
macro, and checks if the macro is set when headers provided by external
libraries are included.
With this, we can fail fast at compile time if necessary `_cflags`
dependencies are omitted in meson.build. Missing dependencies found by this
mechanism have been added across the tree.
This introduce a vtable for journal tagging feature in sd-journal,
and makes libgcrypt dependent features loaded by users (journald,
journalctl, journal-remote, and unit tests) when necessary.
request_handler() owns the hostname var and passes it by value to
request_meta(), which hands it to source_new(), which stores it in
source->importer.name without copying. If build_accept_encoding()
then fails, the hostname var is freed, and then the caller's
_cleanup_free_ frees it a second time.
Follow-up for 9ff48d0982
request_reader_entries() negated m->n_skip in signed context before
casting to uint64_t, which is undefined behaviour for
n_skip == INT64_MIN.
Follow-up for 77ad3b93de
Follow-up for a7bfb9f76b
Switch the executable-owned call sites from the plain dlopen_<lib>()
helpers to the new DLOPEN_<LIB>() macros, so that each binary that loads
an optional library at runtime now carries its own .note.dlopen note
instead of relying solely on libsystemd-shared. Each call site passes a
priority reflecting whether it hard-depends on the library (required) or
degrades gracefully without it (recommended).
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
- **bpf-util: rename from bpf-dlopen, unify version-specific symbol
handling**
- **cryptsetup: dlopen libcryptsetup in tokens**
- **tree-wide: dlopen libpam in pam plugins**
- **test-bus-marshal: dlopen() glib and libdbus instead of linking
directly**
- **lock-util: Simplify timeout for lock_generic_with_timeout()**
- **color-util: simplify hsv_to_rgb, fix rgb_to_hsv negative-hue wrap**
- **tree-wide: Use our own macros instead of fabs()/fmax()/fmin()**
- **locale-util: dlopen() libintl instead of linking against it**
- **home: Use log2u64() over log2()**
- **meson: drop libdl, threads, and librt dependencies**
- **libc: Use dlsym() from a constructor instead of weak symbols**
- **libc: Make sure C23 versions of strtol(), sscanf() are not used**
Our baseline glibc is 2.34, which merged libdl, libpthread (the
dependency('threads') target), and librt into libc. Empty .so/.a stubs
remain for backward compatibility with old binaries, but new builds
resolve dl_*, pthread_*, mq_*, timer_*, etc. directly from libc.
On musl the same libraries are likewise empty stubs.
Drop the libdl, threads, and librt entries from every meson.build, and
remove the now-stale 'Libs.private: -lrt -pthread' from libudev.pc.in
since both flags resolve to empty link-time stubs on glibc 2.34+ and
musl.
Verified with readelf -d that libsystemd.so, libudev.so, and systemd no
longer carry DT_NEEDED entries for libdl/libpthread/librt.
I am really not a fan of full code lines passed to macros as parameters.
Let's get rid of the 3rd parameter of FOREACH_OPTION() hence:
1. Let's return errors just as a regular value (though a negative one),
that can be handled via a OPTION_ERROR case statement for the switch.
This normalizes handling of the error, just like any other event
returned by the option parser.
2. In order to avoid exploding the amount of boilerplate in each use
(that just propagates the error on OPTION_ERROR), let's then
introduce an explicit FOREACH_OPTION_OR_RETURN(), that returns from
the calling function on its own (and makes that clear in the name).
Together this cleans up, normalizes the logic and shortens the code.
It was requested to make the 'c', 'opt', and 'arg' params the same, i.e.
defined through the FOREACH_OPTION macro. But we can't do that easily,
because 'c' was defined in the for loop definition, and we can only
define variables of the same type in that way. Also, in some cases we
need only 'c', in other cases with need 'c' and 'arg, in some cases 'c'
and 'opt', and in other cases all three. We'd need to either
conditionalize or mark those variables with _unused_ to deal with
compiler warnings. But a different approach works quite nicely: add
state.opt and state.arg to show the current option and it's argument.
(The short names are picked on purpose to reduce verbosity since those
are used a lot.)
Convert the GnuTLS and libmicrohttpd usage in journal-remote to the
dlopen pattern used by other optional shared libraries. A new
src/shared/gnutls-util.{h,c} declares the GnuTLS entry points via
DLSYM_PROTOTYPE and resolves them in dlopen_gnutls(); microhttpd-util
is moved from src/journal-remote to src/shared and gains analogous
DLSYM_PROTOTYPEs plus dlopen_microhttpd(). Callers in journal-gatewayd,
journal-remote-main and microhttpd-util itself call the sym_* wrappers
and invoke dlopen_gnutls()/dlopen_microhttpd() at their entry points.
setup_gnutls_logger() no longer fails when libgnutls is missing at
runtime; it logs a notice and returns 0 so journal-gatewayd starts up
without TLS dependencies installed.
The meson files gain libgnutls_cflags and libmicrohttpd_cflags partial
dependencies that expose include paths and compile flags only. Every
systemd-journal-{gatewayd,remote,upload} target switches to the cflags
variant, dropping the direct libgnutls/libmicrohttpd link. The
gatewayd->remote object-sharing dance for microhttpd-util.o goes away
since the code now lives in libshared.
test-dlopen-so gains assertions for dlopen_gnutls and dlopen_microhttpd.
Add options to vmspawn to configure journal-remote disk usage limits
when forwarding journal entries from the VM. These are passed through
as --max-use=, --keep-free=, --max-file-size=, and --max-files=
command-line arguments to systemd-journal-remote.
Add --max-use=, --keep-free=, --max-file-size=, and --max-files=
command-line options to systemd-journal-remote to allow overriding the
corresponding settings from the configuration file.
Add $SYSTEMD_JOURNAL_REMOTE_CONFIG_FILE environment variable support
to systemd-journal-remote. When set, the specified file is used
instead of the default configuration file and drop-in directories.
When set to the empty string or /dev/null, configuration file parsing
is skipped entirely. vmspawn sets this to /dev/null in the child
process to avoid inheriting the host's journal-remote configuration.
Make fork_notify() argv parameter optional. When NULL is passed,
fork_notify() returns 0 in the child (with $NOTIFY_SOCKET set) and
lets the caller run custom code before exec. Returns 1 in the parent.
This allows vmspawn to set environment variables in the child without
polluting the parent process.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace the getopt_long()-based parser with the FOREACH_OPTION /
OPTION_* macros from src/shared/options.h, mirroring the recent
conversions of nspawn and vmspawn. Each option's metadata (long
name, short name, metavar and help text) now lives next to its
parsing logic, and the --help text is generated from those
definitions via option_parser_get_help_table() instead of being
hard-coded.
Positional file arguments are collected via
option_parser_get_args() rather than strv_skip(argv, optind).
Every dlopen_xxx() helper now takes an int log_level argument. It is
passed through to dlopen_many_sym_or_warn() (which in turn propagates it
to dlopen_verbose() for the library-not-installed case), and is used by
the fallback stub when support for the library is not compiled in to
emit a "<lib> support is not compiled in." message at the caller's level.
Callers pass LOG_DEBUG when gracefully degrading, or a higher level when
the failure should surface, and no longer need to log redundantly at the
call site.
As part of this, dlopen_bpf_full() (which already took a log_level) is
merged into dlopen_bpf() rather than keeping both.
The static inline fallbacks used to live in the headers, which required
pulling log.h in from every header that declared a dlopen_xxx(). Move
them into the .c files instead: the declaration is always outside the
#if HAVE_XXX block, the impl sits outside the outer #if HAVE_XXX wrap
with its own internal #if HAVE_XXX/#else/#endif, and apparmor-util.c,
idn-util.c, libmount-util.c and pam-util.c are now always compiled so
they can host their stubs.
Convert curl-util to the dlopen pattern used by other optional shared
libraries in libshared (libarchive, pcre2, idn, ...). Declare the curl
API entry points with DLSYM_PROTOTYPE, resolve them in a dlopen_curl()
helper, and call the sym_* wrappers from callers. curl_glue_new() now
loads the library on first use, so consumers going through CurlGlue
pick this up automatically; journal-upload and report-upload call
dlopen_curl() directly since they use curl without the glue layer.
With this in place curl-util can live in libshared itself, linked only
against libcurl's headers (via libcurl_cflags). The libcurlutil_static
convenience library and the libcurl link dependency on systemd-imdsd,
systemd-pull, systemd-journal-upload and systemd-report go away.
Also move the easy_setopt() helper macro next to the DLSYM declarations
so all consumers use a single sym-prefixed definition, and add a
dlopen_curl() check to test-dlopen-so.
RFC 8996 says:
> This document formally deprecates Transport Layer Security (TLS)
> versions 1.0 (RFC 2246) and 1.1 (RFC 4346). Accordingly, those
> documents have been moved to Historic status. These versions lack
> support for current and recommended cryptographic algorithms and
> mechanisms, and various government and industry profiles of
> applications using TLS now mandate avoiding these old TLS versions.
> TLS version 1.2 became the recommended version for IETF protocols in
> 2008 (subsequently being obsoleted by TLS version 1.3 in 2018),
> providing sufficient time to transition away from older versions.
> Removing support for older versions from implementations reduces the
> attack surface, reduces opportunity for misconfiguration, and
> streamlines library and product maintenance.
This code probably only talks to our own receiver which uses
libmicrohttpd. That in turn delegates to GnuTLS, which supports
1.2, 1.3, 3.0, etc.
Move more common definitions in the header file instead of repeating
them in bunch of places. src/import/curl-util.[ch] is renamed so that
it's shared more naturally with other components.
We cannot use a function, because the type is unknown and we want
to stringify the option name, but we can use a block macro to make
this a bit nicer, with normal code structure in the caller.
When --trust=all disables CURLOPT_SSL_VERIFYPEER, the residual
CURLOPT_SSL_VERIFYHOST check is ineffective since an attacker can
present a self-signed certificate with the expected hostname. Disable
both for consistency and log that server certificate verification is
disabled.
Follow-up for 8847551bcb
I promised in https://github.com/systemd/systemd/pull/41426 that its the
final update for coccinelle pointer deref checks. However it turned out
there is this coccinelle/parsing_hacks.h that I wasn't aware of. The
file missed some important things like _cleanup_(x) that prevented
coccinelle to check a bunch of functions.
This PR adds some missing defines to the parsing_hacks.h and fixes the
missing asserts(). I apologize that its a bit long (and frankly boring)
and that I missed this earlier.
The last commit contains one small behavior change (ret in
sd_varlink_idl_parse() is now really optional) but the big one is very
mechanical.
We already have checks in place during compression that limit the data
we compress, so they shouldn't decompress to anything larger than
DATA_SIZE_MAX unless they've been tampered with. Let's make this
explicit and limit all our decompress_blob() calls in journal-handling
code to that limit.
One possible scenario this should prevent is when one tries to open and
verify a journal file that contains a compression bomb in its payload:
$ ls -lh test.journal
-rw-rw-r--+ 1 fsumsal fsumsal 1.2M Apr 12 15:07 test.journal
$ systemd-run --user --wait --pipe -- build-local/journalctl --verify --file=$PWD/test.journal
Running as unit: run-p682422-i4875779.service
000110: Invalid hash (00000000 vs. 11e4948d73bdafdd)
000110: Invalid object contents: Bad message
File corruption detected at /home/fsumsal/repos/@systemd/systemd/test.journal:272 (of 1249896 bytes, 0%).
FAIL: /home/fsumsal/repos/@systemd/systemd/test.journal (Bad message)
Finished with result: exit-code
Main processes terminated with: code=exited, status=1/FAILURE
Service runtime: 48.051s
CPU time consumed: 47.941s
Memory peak: 8G (swap: 0B)
Same could be, in theory, possible with just `journalctl --file=`, but
the reproducer would be a bit more complicated (haven't tried it, yet).
Lastly, the change in journal-remote is mostly hardening, as the maximum
input size to decompress_blob() there is mandated by MHD's connection
memory limit (set to JOURNAL_SERVER_MEMORY_MAX which is 128 KiB at the
time of writing), so the possible output size there is already quite
limited (e.g. ~800 - 900 MiB for xz-compressed data).
The updated parsing_hacks.h file uncovered a bunch of extra
things that the check-pointer-deref coccinelle script flags.
This commit fixes them to make the tree check-pointer-deref clean.
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
The fix in 8f1751a111 made me wonder if we could automatically detect
when pointers are accessed but when this might not be safe. Systemd
is already using a lot of `assert(dst)` and this change now forces
us to use them.
So this commit (ab)uses coccinelle to flag any pointer parameter
dereference not preceded by assert(param), ASSERT_PTR(param), or an
explicit NULL check. It adds integration into meson as a new "coccinelle"
test suite (just like clang-tidy) and is run in CI. The check is not
perfect but seems a reasonable heuristic.
For this RFC commit it is scoped to a subset, it excludes 25 dirs right
now and includes around 100. About 300 warnings left. Busywork that I am
happy to do if there is agreement that it is worth it.
With this in place we would have caught the bug from 8f1751a111 in CI:
```
FAIL: check-pointer-deref.cocci found issues in systemd/src/boot:
diff -u -p systemd/src/boot/measure.c /tmp/nothing/measure.c
--- systemd/src/boot/measure.c
+++ /tmp/nothing/measure.c
@@ -312,7 +312,6 @@ EFI_STATUS tpm_log_tagged_event(
if (err != EFI_SUCCESS)
return err;
- *ret_measured = true;
return EFI_SUCCESS;
}
```
This also adds a new POINTER_MAY_BE_NULL() for the cases when the
called function will do the NULL check (like `iovec_is_set()`).
The line to set opterr=0 was added in the initial commit in
3d090cc6f3. But afaict, this never worked as
intended, because ':' must be the first char in optstring given to
getopt_long() for it to return ':' for a missing option value. Since
this wasn't set, getopt_long() would return '?', and the missing value
would be handled as an unknown option:
$ build/systemd-journal-upload --key
Unknown option --key.
$ build/systemd-journal-upload --asdf
Unknown option --asdf.
Let's just do the standard thing:
$ build/systemd-journal-upload --key
build/systemd-journal-upload: option '--key' requires an argument
$ build/systemd-journal-upload --asdf
build/systemd-journal-upload: unrecognized option '--asdf'
felix h
Zbigniew Jędrzejewski-Szmek
Yu Watanabe
Luca Boccassi
Christian Hesse
Daan De Meyer
Lennart Poettering
Frantisek Sumsal
Michael Vogt
Zbigniew Jędrzejewski-Szmek
Lennart Poettering
Daan De Meyer