Since all executables now manage their required dlopen notes directly
within their own source code with explicit priority levels, it is no
longer necessary to declare high-priority dlopen notes in the shared
libraries themselves.
Previously, even though sd-journal does not support bzip2 and gzip
compression, the dlopen ELF notes for those libraries were still being
attached to libsystemd.so and various journal-handling executables.
This occurred because the unified compression/decompression interfaces
handled all formats unconditionally, causing the compiler and linker to
pull in all associated dlopen notes across the board.
To resolve this, split these functions into generic and journal-specific
variants (e.g., introducing compress_blob_journal() and decompress_blob_journal()).
The journal-specific variants only handle formats actually supported by
the journal (LZ4, XZ, and ZSTD).
By updating sd-journal, journald, journalctl, and related utilities to
use these new `_journal` interfaces and switching them to the narrower
COMPRESS_JOURNAL_NOTE macro, we ensure that unnecessary dlopen notes
(for bzip2 and gzip) are no longer embedded into these binaries.
When compiler optimization or LTO (Link Time Optimization) is enabled,
dlopen helper functions (such as dlopen_libfoo()) can be aggressively
inlined into their callers or cloned for specific call sites.
While this is beneficial for production builds, it alters or completely
removes the original function symbols from the final executable.
Consequently, this breaks the test (to be added in this series) that
checks whether the dlopen_libfoo() function corresponding to a dlopen
ELF note is actually called.
To resolve this, introduce the `_dlopen_loader_` macro. Under developer
builds, it applies the `_noclone_` and `_noinline_` attributes to guarantee
that helper symbols remain intact and discoverable by the test suite. In
production builds, the macro evaluates to nothing, allowing full compiler
optimization to proceed unimpeded.
This also switches all these notes to be defined via SD_ELF_NOTE_DLOPEN_ANCHORED().
Consequently, any notes added within unreachable or unused functions will be
automatically garbage-collected by the linker (--gc-sections) instead of
bloating the final binary.
E.g. unused p11-kit library dependency is now dropped from
systemd-repart.standalone binary.
Before:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
idn Support for internationalized domain names libidn2.so.0 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
p11-kit Support for PKCS11 hardware tokens libp11-kit.so.0 suggested
```
After:
```
$ systemd-analyze dlopen-metadata build/systemd-repart.standalone
FEATURE DESCRIPTION SONAME PRIORITY
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 recommended
blkid Support for block device identification libblkid.so.1 required
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 recommended
mount Support for mount enumeration libmount.so.1 required
fdisk Support for reading and writing partition tables libfdisk.so.1 required
blkid Support for block device identification libblkid.so.1 recommended
libcrypto Support for cryptographic operations libcrypto.so.4 libcrypto.so.3 suggested
cryptsetup Support for disk encryption, integrity, and authentication libcryptsetup.so.12 suggested
fdisk Support for reading and writing partition tables libfdisk.so.1 suggested
mount Support for mount enumeration libmount.so.1 recommended
selinux Support for SELinux libselinux.so.1 recommended
tpm Support for TPM libtss2-esys.so.0 suggested
tpm Support for TPM libtss2-rc.so.0 suggested
tpm Support for TPM libtss2-mu.so.0 suggested
tpm Support for TPM libtss2-tcti-device.so.0 suggested
```
This makes each `foo_cflags` dependency define a `SYSTEMD_CFLAGS_MARKER_FOO`
macro, and checks if the macro is set when headers provided by external
libraries are included.
With this, we can fail fast at compile time if necessary `_cflags`
dependencies are omitted in meson.build. Missing dependencies found by this
mechanism have been added across the tree.
The zstd blob decompression code assumed ZSTD_getFrameContentSize() always
returns the decompressed size, which holds for the journal and coredump blobs
systemd compresses itself with the one-shot ZSTD_compress(). Frames produced by
the streaming API, however, don't record the decompressed size in their header,
so ZSTD_getFrameContentSize() returns ZSTD_CONTENTSIZE_UNKNOWN. Per the zstd
documentation this is not an error, but decompress_blob_zstd() and
decompress_startswith_zstd() bailed out with -EBADMSG regardless.
This broke kexec of kernel images compressed with 'zstd' (e.g. zstd -22), whose
ZBOOT payload is decompressed via decompress_blob().
Treat only ZSTD_CONTENTSIZE_ERROR as fatal and, when the size is unknown, grow
the output buffer as we stream the frame out instead of relying on the recorded
size. Add a regression test that feeds a streaming-compressed (hence
unknown-size) zstd frame through decompress_blob() and decompress_startswith().
Each dlopen_*() wrapper kept its dl handle as a file-scope
'static void *xxx_dl = NULL;' even though only the wrapper itself
ever referenced it. Move each one inside the corresponding function
so its scope matches its actual use, leaving the rest of each
translation unit free of the unused file-scope name.
In pcre2-util.c the assert(pcre2_dl) in pattern_matches_and_log()
becomes assert(sym_pcre2_match), which carries the same invariant
(pattern_compile_and_log() ran dlopen_pcre2()).
Every dlopen_xxx() helper now takes an int log_level argument. It is
passed through to dlopen_many_sym_or_warn() (which in turn propagates it
to dlopen_verbose() for the library-not-installed case), and is used by
the fallback stub when support for the library is not compiled in to
emit a "<lib> support is not compiled in." message at the caller's level.
Callers pass LOG_DEBUG when gracefully degrading, or a higher level when
the failure should surface, and no longer need to log redundantly at the
call site.
As part of this, dlopen_bpf_full() (which already took a log_level) is
merged into dlopen_bpf() rather than keeping both.
The static inline fallbacks used to live in the headers, which required
pulling log.h in from every header that declared a dlopen_xxx(). Move
them into the .c files instead: the declaration is always outside the
#if HAVE_XXX block, the impl sits outside the outer #if HAVE_XXX wrap
with its own internal #if HAVE_XXX/#else/#endif, and apparmor-util.c,
idn-util.c, libmount-util.c and pam-util.c are now always compiled so
they can host their stubs.
If a journal file contains a truncated ZSTD frame (i.e. a frame with
Frame_Content_Size > 0, but with not enough data in Data_Block),
ZSTD_decompressStream() would return a non-zero, non-error value. This
would then skip the error path in the ZSTD_isError() branch and we'd hit
the following assert:
$ build-local/journalctl -o cat --file zstd-truncated.journal
Assertion 'output.pos >= prefix_len + 1' failed at src/basic/compress.c:1236, function decompress_startswith_zstd(). Aborting.
Aborted (core dumped) build-local/journalctl -o cat --file zstd-truncated.journal
Let's handle this situation gracefully and return EBADMSG instead.
Also, add another journalctl invocation to the corrupted-journals test
that goes through the sd_journal_get_data() -> decompress_startswith_zstd()
code path which, among other things, covers the issue when run on the
provided journal file.
We already have checks in place during compression that limit the data
we compress, so they shouldn't decompress to anything larger than
DATA_SIZE_MAX unless they've been tampered with. Let's make this
explicit and limit all our decompress_blob() calls in journal-handling
code to that limit.
One possible scenario this should prevent is when one tries to open and
verify a journal file that contains a compression bomb in its payload:
```
$ ls -lh test.journal
-rw-rw-r--+ 1 fsumsal fsumsal 1.2M Apr 12 15:07 test.journal
$ systemd-run --user --wait --pipe -- build-local/journalctl --verify --file=$PWD/test.journal
Running as unit: run-p682422-i4875779.service
000110: Invalid hash (00000000 vs. 11e4948d73bdafdd)
000110: Invalid object contents: Bad message
File corruption detected at /home/fsumsal/repos/@systemd/systemd/test.journal:272 (of 1249896 bytes, 0%).
FAIL: /home/fsumsal/repos/@systemd/systemd/test.journal (Bad message)
Finished with result: exit-code
Main processes terminated with: code=exited, status=1/FAILURE
Service runtime: 48.051s
CPU time consumed: 47.941s
Memory peak: 8G (swap: 0B)
```
Same could be, in theory, possible with just `journalctl --file=`, but
the reproducer would be a bit more complicated (haven't tried it, yet).
Lastly, the change in journal-remote is mostly hardening, as the maximum
input size to decompress_blob() there is mandated by MHD's connection
memory limit (set to JOURNAL_SERVER_MEMORY_MAX which is 128 KiB at the
time of writing), so the possible output size there is already quite
limited (e.g. ~800 - 900 MiB for xz-compressed data).
Replace the duplicated magic byte signatures in decompressor_detect()
with a call to the new compression_detect_from_magic() helper and use a
switch statement to initialize the appropriate decompression context.
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Core dumps are often very sparse, containing large zero-filled regions
whose actual disk usage can be significantly reduced by preserving
holes. Previously, decompress_stream() always wrote dense output,
expanding all zero regions into allocated disk blocks.
Each decompression backend (xz, lz4, zstd) now auto-detects whether the
output fd is suitable for sparse writes via a shared should_sparse()
helper. The check requires both S_ISREG (regular file) and !O_APPEND,
since O_APPEND causes write() to ignore the file position set by
lseek(), which would collapse the holes and corrupt the output. For
pipes, sockets, and append-mode files, dense writes are preserved via
loop_write_full() with USEC_INFINITY timeout, matching the original
behavior. After sparse decompression, finalize_sparse() sets the final
file size to account for any trailing holes.
This is transparent to callers — all public signatures are unchanged.
coredumpctl benefits automatically:
- coredumpctl debug: temp file in /var/tmp is now sparse
- coredumpctl dump -o file: output file is now sparse
- coredumpctl dump > file: redirected stdout is now sparse
- coredumpctl dump | ...: pipe output unchanged (dense)
- coredumpctl dump >> file: append mode, falls back to dense
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-developed-by: Codex (GPT-5) <noreply@openai.com>
Coverity flags 2 * space as a potential overflow. The space value
is bounded by prior allocation success, but add an explicit assert
to document this for static analyzers.
CID#1548056
Follow-up for 5e592c66bd
Switch all internal callers from the private ELF_NOTE_DLOPEN() macro to
the new public SD_ELF_NOTE_DLOPEN() API from sd-dlopen.h, and remove
the now-redundant macro definitions from dlfcn-util.h.
Signed-off-by: Christian Brauner <brauner@kernel.org>
Suppress warnings like the following from clang tidy:
```
../src/boot/addon.c:11:19: error: function 'efi_main' can be made static to enforce internal linkage [misc-use-internal-linkage,-warnings-as-errors]
11 | EFIAPI EFI_STATUS efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *system_table);
| ^
```
Some warnings are suppressed simply by setting comments to ignore the warning,
some are by making global variables static, or include a suitable header.
If the first call to `loop_read()` returns 0 (no input), `total_in`
remains 0, causing `total_out/total_in` to potential divide by zero.
We add a check before logging the compression ratio to skip the
percentage calculation when total_in is zero.
Co-authored-by: jinyaoguo <guo846@purdue.edu>
If zstd frames are corrupted the initial size returned for the current
frame might be wrong. Don#t assert() on that, but handle it gracefully,
as EBADMSG
The lz4 functions are only used in test-compress.c, so let's just
put the declarations and includes in there instead of having everyone
including compress.h pull in the lz4 headers.
Now that the necessary functions from log.h have been moved to macro.h,
we can stop including log.h in macro.h. This requires modifying source
files all over the tree to include log.h instead.
When looking at how dlopen for various libs is implemented, I found that the
macros hide too much. I find it much easier to see what is going on if 'extern'
and '= NULL' are written explicitly. After all, we don't hide those for other
definitions, e.g. our style guide says that static variables should be
initialized with '= NULL'. With that change, it's much more obvious what is
a variable declaration and what is a variable initialization.
Use 'recommended' priority for the default compression library, to
indicate that it should be prioritized over the other ones, as it
will be used to compress journals/core files.
Also use 'recommended' for kmod, as systems will likely fail to boot
if it's missing from the initrd.
Use 'suggested' for everything else.
There is one dlopen'ed TPM library that has the name generated
at runtime (depending on the driver), so that cannot be added, as it
needs to be known at build time.
Also when we support multiple ABI versions list them all, as for the
same reason we cannot know which one will be used at build time.
$ dlopen-notes.py build/libsystemd.so.0.39.0 build/src/shared/libsystemd-shared-256.so
libarchive.so.13 suggested
libbpf.so.0 suggested
libbpf.so.1 suggested
libcryptsetup.so.12 suggested
libdw.so.1 suggested
libelf.so.1 suggested
libfido2.so.1 suggested
libgcrypt.so.20 suggested
libidn2.so.0 suggested
libip4tc.so.2 suggested
libkmod.so.2 recommended
liblz4.so.1 suggested
liblzma.so.5 suggested
libp11-kit.so.0 suggested
libpcre2-8.so.0 suggested
libpwquality.so.1 suggested
libqrencode.so.3 suggested
libqrencode.so.4 suggested
libtss2-esys.so.0 suggested
libtss2-mu.so.0 suggested
libtss2-rc.so.0 suggested
libzstd.so.1 recommended
Co-authored-by: Luca Boccassi <bluca@debian.org>
Dynamically load liblz4, libzstd and liblzma with dlopen().
This helps to reduce the size of the initrd image when these libraries
are not really needed.
Given we have two different types for the journal object flags and the
Compression enum, let's make the latter a regular non-sparse enum, and
thus remove some surprises. We have to convert anyway between the two,
and already do via COMPRESSION_FROM_OBJECT().
The compression helpers are used both in journal code and in coredump
code, and there's a good chance we'll use them later for other stuff.
Let's hence move them into src/basic/, to make them a proper internal
API we can use from everywhere where that's desirable. (pstore might be
a candidate, for example)
No real code changes, just some moving around, build system
rearrangements, and stripping of journal-def.h inclusion.
Yu Watanabe
Daan De Meyer
Frantisek Sumsal
Zbigniew Jędrzejewski-Szmek
noxiouz
Luca Boccassi
Christian Brauner
Daan De Meyer
Lennart Poettering
Alexandjinyaoguo
Andrii Chubatiuk
Mike Yuan
Zbigniew Jędrzejewski-Szmek
Matteo Croce