Previously, we'd try to handle corruption by bumping the index even
if it was an entry array object that was corrupted (which we can't
deal with).
Now, we only try to deal with corrupted entry objects by moving the
corruption handling into generic_array_get().
On top, we also add an additional check for -EADDRNOTAVAIL which can
also be caused by corrupted journal data.
This patch changes busctl capture to generate pcapng format
instead of the legacy pcap format files. It includes basic
meta-data in the file and still uses microsecond time
resolution. In future, more things can be added such as
high resolution timestams, statistics, etc.
PCAP Next Generation capture file format is what tshark uses
and is in process of being standardized in IETF. It is also
readable with libpcap.
$ capinfos /tmp/new.pcapng
File name: /tmp/new.pcapng
File type: Wireshark/... - pcapng
File encapsulation: D-Bus
File timestamp precision: microseconds (6)
Packet size limit: file hdr: (not set)
Packet size limit: inferred: 4096 bytes
Number of packets: 22
File size: 21kB
Data size: 20kB
Capture duration: 0.005694 seconds
First packet time: 2021-12-11 11:57:42.788374
Last packet time: 2021-12-11 11:57:42.794068
Data byte rate: 3,671kBps
Data bit rate: 29Mbps
Average packet size: 950.27 bytes
Average packet rate: 3,863 packets/s
SHA256: b85ed8b094af60c64aa6d9db4a91404e841736d36b9e662d707db9e4096148f1
RIPEMD160: 81f9bac7ec0ec5cd1d55ede136a5c90413894e3a
SHA1: 8400822ef724b934d6000f5b7604b9e6e91be011
Strict time order: True
Capture oper-sys: Linux 5.14.0-0.bpo.2-amd64
Capture application: systemd 250 (250-rc2-33-gdc79ae2+)
Number of interfaces in file: 1
Interface #0 info:
Encapsulation = D-Bus (146 - dbus)
Capture length = 4096
Time precision = microseconds (6)
Time ticks per second = 1000000
Number of stat entries = 0
Number of packets = 22
The meson default for static_library() are:
build_by_default=true, install=false. We never interact with the
static libraries, and we only care about them as a stepping-stone towards
the installable executables or libraries. Thus let's only build them if
they are a dependency of something else we are building.
While at it, let's drop install:false, since this appears to be the default.
This change would have fixed the issue with lib_import_common failing
to build too: we wouldn't attempt to build it.
In practice this changes very little, because we generally only declare static
libraries where there's something in the default target that will make use of
them. But it seems to be a better pattern to set build_by_default to false.
The variable is not useful outside of the loop (it'll always be null
after the loop is finished), so we can declare it inline in the loop.
This saves one variable declaration and reduces the chances that somebody
tries to use the variable outside of the loop.
For consistency, 'de' is used everywhere for the var name.
Previously the MMapCache* was optionally NULL, which open would
handle by creating a new MMapCache* for the occasion.
This produced some slightly circuitous refcount-handling code in
the function, as well as arguably creating opportunities for
weirdness where an MMapCache* was intended to be supplied but
happened to be NULL, which this magic would then paper over.
In any case, this was basically only being utilized by tests,
apparently just to avoid having to create an MMapCache. So
update the relevant tests to supply an MMapCache and make
journal_file_open() treat a NULL MMapCache* as fatal w/assert.
This gets rid of the manual MMapCache ref/unref goop in
journal_file_{open,close}(), in favor of just letting the
JournalFile.cache_fd MMapFileDescriptor carry the baton.
Preparatory commit; before JournalFile can stop hanging onto its
copy of MMapCache, all these users need to find another way.
Most of the time these callers already have the MMapCache onhand,
so it's no big deal for them to just supply it.
journal_file_rotate() in particular needed to change, and it
seemed wise to not use the mmap_cache_fd_cache() accessor on
f->cache_fd, instead requiring the caller supply the cache to
use. This was done with an eye towards a potential future where
the journal_file_archive() isolates the cache_fd to a private
cache, which the newly rotated-to file wouldn't be allowed to
use. It's no biggie for the existing callers to just provide the
appropriate surviving cache.
Basically the mmap_cache_fd_cache() accessor was added just for
journal-verify.c's (ab)use of the mmap-cache. Which, if the
ugly singleton MMapCache assumption ever goes away, can be
cleaned up to simply use a separate MMapCache for those search
arrays.
Preparatory commit; callers manually ref/unref MMapCaches
alongside MMapFileDescriptor add/frees, when the latter should be
sufficient.
A subsequent commit will drop some of those manual MMapCache
reference hoop-jumping, leaving the lifecycle bound to
MMapFileDescriptors.
Disabling NOCOW when data has been written to a file doesn't work.
Instead, when we're done writing to a journal file (after archiving),
let's rewrite the file with COW enabled. This also takes care of
properly defragmenting the file.
With zstd compression level 3, journal files are compressed to 12%
of their original size with default journal settings.
As rewriting the file might take a while since we also do an fsync()
after the rewrite, this work is done in the offline thread to avoid
blocking the journald event loop.
When we archive a path, we rename the file to indicate this. However,
until now, we didn't actually update the path member of the corresponding
JournalFile instance. Let's make sure we also update this to avoid
misuse of the old path later on.
This change also requires we save the previous path in journal_file_rotate()
since we need to open a new file at the previous path.
With this change, the logic to write the final tag, emit the final
change notification and to offline the file moves from journal_file_close()
to journald_file_close(). Since all this logic is only executed when
the journal file is writable and all code that writes journal files
already uses journald_file_close() instead of journal_file_close(), this
change should not introduce any changes in behaviour.
Moving the offline related logic to journald-file.c allows us to use
code from src/shared in the offlining logic, more specifically, we can
use the file copying logic from copy.h to fix BTRFS filesystem compression
for journal files when archiving.
Currently, all the logic related to writing journal files lives in
journal-file.c which is part of libsystemd (sd-journal). Because it's
part of libsystemd, we can't depend on any code from src/shared.
To allow using code from src/shared when writing journal files, let's
gradually move the write related logic from journal-file.c to
journald-file.c in src/journal. This directory is not part of libsystemd
and as such can use code from src/shared.
We can safely remove any journal write related logic from libsystemd as
it's not used by any public APIs in libsystemd.
This commit introduces the new file along with the JournaldFile struct
which wraps an instance of JournalFile. The goal is to gradually move
more functions from journal-file.c and fields from JournalFile to
journald-file.c and JournaldFile respectively.
This commit also modifies all call sites that write journal files to
use JournaldFile instead of JournalFile. All sd-journal tests that
write journal files are moved to src/journal so they can make use of
journald-file.c.
Because the deferred closes logic is only used by journald, we move it
out of journal-file.c as well. In journal_file_open(), we would wait for
any remaining deferred closes for the file we're about to open to complete
before continuing if the file was not newly created. In journald_file_open(),
we call this logic unconditionally since it stands that if a file is newly
created, it can't have any outstanding deferred closes.
No changes in behavior are introduced aside from the earlier execution
of waiting for any deferred closes to complete when opening a new journal
file.
Entry arrays grow exponentially, so when archiving a journal file is
archived, it's very likely that the final entry array objects in each
entry array chain aren't fully used. Let's punch holes in the unused
parts so the filesystem can reclaim this unused space and use it for
something else.
Journal files have space allocated in 8MiB-aligned increments. This
can add up to substantial wasted space as many archived journals
accumulate without using all the allocated space. Let's truncate
journal files to their actually used size when archiving them to
reclaim this unused space.
As the mmap cache is not thread-safe, we can't call
journal_file_move_to_object() from the offline thread. Instead,
we use journal_file_read_object() which doesn't rely on the mmap
cache.
We would fail if the input file was corrupted:
build/test-journal-flush ./system@0005b7dac334f805-0021aca076ae5c5e.journal\~
journal_file_copy_entry failed: Bad message
Assertion 'r >= 0' failed at src/libsystemd/sd-journal/test-journal-flush.c:55, function main(). Aborting.
[1] 619472 IOT instruction (core dumped) build/test-journal-flush ./system@0005b7dac334f805-0021aca076ae5c5e.journal\~
Let's skip some "reasonable" errors.
Fixes#17963.
Code simplification; no idea why the Context structs were being
heap-allocated piecemeal. There aren't many and they're small,
and made even smaller with this commit.
This also gets rid of the Context.cache backreference by just
accompanying the Context* with MMapCache* to the few private
Context* operations, resolving the context ID to a Context* at
the single public entrypoint: mmap_cache_fd_get(), which asserts
on the value being within bounds. The Context.id member and
checks are also removed, as there's far less opportunity for
these things to become inconsistent now.
Of course this drops all context_{add,free}() related code,
making for a tasty diffstat.
If the iovec contains the same data more than once, we'll end up with
duplicate offsets in the items array. Let's make sure we remove any
duplicates before we store the items in an entry object.
The LIST_REMOVE() macro always assigns NULL to w->unused_prev,
meaning every time this window was in last_unused, the remainder
of the unused list was lost to the ether.
Turns out there's been a memory leak in journald after all, this
code has been there since at least 2013...
This actually causes mismatches between the header tail entry seqnum
and the last entry seqnum since when we revert the header seqnum, we
don't remove the entry object we added. If adding the entry object
itself fails, we don't need to revert the seqnum since it's never
incremented so let's remove this logic alltogether.
Yu Watanabe
Daan De Meyer
Jan Janssen
Zbigniew Jędrzejewski-Szmek
Stephen Hemminger
Luca Boccassi
Vito Caputo
Lennart Poettering