Previously, even when the compressed data was truncated, or had
trailing garbage, the decompressor might not detect it.
To fix the issue, update the check if stream is finished.
With this change, now decompressor_push() must be called with
size == 0 to finalize the stream.
Otherwise, random input cases are not tested:
```
/* test_compress_decompress_blob */
/* testing xz text blob compression/decompression */
Loaded shared library 'liblzma.so.5' via dlopen().
/* testing xz random blob compression/decompression */
compression failed: No buffer space available
/* testing lz4 text blob compression/decompression */
Loaded shared library 'liblz4.so.1' via dlopen().
/* testing lz4 random blob compression/decompression */
compression failed: No buffer space available
/* testing zstd text blob compression/decompression */
Loaded shared library 'libzstd.so.1' via dlopen().
/* testing zstd random blob compression/decompression */
compression failed: No buffer space available
/* testing gzip text blob compression/decompression */
Loaded shared library 'libz.so.1' via dlopen().
/* testing gzip random blob compression/decompression */
compression failed: No buffer space available
/* testing bzip2 text blob compression/decompression */
Loaded shared library 'libbz2.so.1' via dlopen().
/* testing bzip2 random blob compression/decompression */
compression failed: No buffer space available
```
The zstd blob decompression code assumed ZSTD_getFrameContentSize() always
returns the decompressed size, which holds for the journal and coredump blobs
systemd compresses itself with the one-shot ZSTD_compress(). Frames produced by
the streaming API, however, don't record the decompressed size in their header,
so ZSTD_getFrameContentSize() returns ZSTD_CONTENTSIZE_UNKNOWN. Per the zstd
documentation this is not an error, but decompress_blob_zstd() and
decompress_startswith_zstd() bailed out with -EBADMSG regardless.
This broke kexec of kernel images compressed with 'zstd' (e.g. zstd -22), whose
ZBOOT payload is decompressed via decompress_blob().
Treat only ZSTD_CONTENTSIZE_ERROR as fatal and, when the size is unknown, grow
the output buffer as we stream the frame out instead of relying on the recorded
size. Add a regression test that feeds a streaming-compressed (hence
unknown-size) zstd frame through decompress_blob() and decompress_startswith().
Move the push-based streaming compression API from import-compress.c
into compress.c and delete import-compress.c/h. This consolidates all
compression code in one place and makes all compression libraries
(liblzma, liblz4, libzstd, libz, libbz2) runtime-loaded via dlopen
instead of directly linked.
Introduce opaque Compressor/Decompressor types backed by a heap-
allocated struct defined only in compress.c, keeping all third-party
library headers out of compress.h.
Rewrite the per-codec fd-to-fd stream functions as thin wrappers around
the push API via generic compress_stream()/decompress_stream() taking a
Compression type parameter. Integrate LZ4 into this framework using the
LZ4 Frame API, eliminating all LZ4 special-casing.
Extend the Compression enum with COMPRESSION_GZIP and COMPRESSION_BZIP2
and add the corresponding blob, startswith, and stream functions for
both.
Rename the ImportCompress types and functions: ImportCompressType becomes
the existing Compression enum, ImportCompress becomes Compressor (with
Decompressor typedef), and all import_compress_*/import_uncompress_*
become compressor_*/decompressor_*. Rename dlopen_lzma() to dlopen_xz()
for consistency. Make compression_to_string() return lowercase by
default.
Add INT_MAX/UINT_MAX overflow checks for LZ4, zlib, and bzip2 blob
functions where the codec API uses narrower integer types than our
uint64_t parameters.
Migrate test-compress.c and test-compress-benchmark.c to the TEST()
macro framework, new assertion macros, and codec-generic loops instead
of per-codec duplication.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Core dumps are often very sparse, containing large zero-filled regions
whose actual disk usage can be significantly reduced by preserving
holes. Previously, decompress_stream() always wrote dense output,
expanding all zero regions into allocated disk blocks.
Each decompression backend (xz, lz4, zstd) now auto-detects whether the
output fd is suitable for sparse writes via a shared should_sparse()
helper. The check requires both S_ISREG (regular file) and !O_APPEND,
since O_APPEND causes write() to ignore the file position set by
lseek(), which would collapse the holes and corrupt the output. For
pipes, sockets, and append-mode files, dense writes are preserved via
loop_write_full() with USEC_INFINITY timeout, matching the original
behavior. After sparse decompression, finalize_sparse() sets the final
file size to account for any trailing holes.
This is transparent to callers — all public signatures are unchanged.
coredumpctl benefits automatically:
- coredumpctl debug: temp file in /var/tmp is now sparse
- coredumpctl dump -o file: output file is now sparse
- coredumpctl dump > file: redirected stdout is now sparse
- coredumpctl dump | ...: pipe output unchanged (dense)
- coredumpctl dump >> file: append mode, falls back to dense
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-developed-by: Codex (GPT-5) <noreply@openai.com>
To make sure everything still compiles, we add a preliminary include
of forward.h to tests.h to make sure it is included in every test source
file. We'll clean up the tests.h includes in a later commit.
We also add a <errno.h> include to errno-list.h to keep test-errno-list.c
compiling. It'll be removed again when we clean up includes in src/basic.
Split out of #37344.
The lz4 functions are only used in test-compress.c, so let's just
put the declarations and includes in there instead of having everyone
including compress.h pull in the lz4 headers.
Dynamically load liblz4, libzstd and liblzma with dlopen().
This helps to reduce the size of the initrd image when these libraries
are not really needed.
The compression helpers are used both in journal code and in coredump
code, and there's a good chance we'll use them later for other stuff.
Let's hence move them into src/basic/, to make them a proper internal
API we can use from everywhere where that's desirable. (pstore might be
a candidate, for example)
No real code changes, just some moving around, build system
rearrangements, and stripping of journal-def.h inclusion.
Yu Watanabe
Daan De Meyer
noxiouz
Daan De Meyer
Andrii Chubatiuk
Unique-Usman
Matteo Croce
Lennart Poettering