IN C23, thread_local is a reserved keyword and we shall therefore
do nothing to redefine it. glibc has it defined for older standard
version with the right conditions.
v2 by Yu Watanabe:
Move the definition to missing_threads.h like the way we define e.g.
missing syscalls or missing definitions, and include it by the users.
Co-authored-by: Yu Watanabe <watanabe.yu+github@gmail.com>
With this we should be able to determine on systems without
battery-backed RTC even during early boot whether a boot is supposed to
be earlier than another.
Fixes: #662
This is useful to later order boot IDs by time, addressing #662.
Basically, this determines the most recently written for each boot ID
from all currently open journal files. This is then stored in a hash
table (which maps the boot ID to a prioq of journal files, ordered by
their timestamp).
Why is this useful? If systems lack a battery-buffered RTC they will
initially have a system clock basically starting at zero. Later they
might acquire an NTP fix, or at least roughly monotonic time via a
stored timestamp. Thus, log entries written early during boot tend to be
badly timestamped, and those written most recently are likely to have
most accurate timestamps. Thus, if we track the newest entry for each
boot ID we likely can order the boot ID via their timestamps.
This commit only add the logic to maintain the hash table/prioq. It
doesn't actually make use of this information for ordering yet. A later
patch adds that.
This way we can quickly find the most recent entry, without searching or
traversing entry array chains.
This is relevant later, as it it allows us to quickly determine the most
recent timestamps of each journal file, in a roughly atomic way.
If CONFIG_PSI_DEFAULT_DISABLED is set in the kernel, then the PSI files
will be there, and you can open them, but read()/write() will fail.
Which is terrible, since that happens so late. But anyway, handle this
gracefully.
This is a follow-up for b1712fabd1 which
was prepped for 253, but merged into early 254 development cycle. It
thus had the symbol it adds at the wrong symver. Fix thta.
If the boot ID cannot be obtained, let's first fallback to the machine
ID, and if still cannot, then let's use 0.
Otherwise, no timer event source cannot be triggered.
Fixes#26549.
In older glibc (like 2.28 on CentOS Stream 8) there is no wrapper
for the gettid() syscall, so we need to provide our own.
../src/libsystemd/sd-journal/journal-send.c: In function ‘close_journal_fd’:
../src/libsystemd/sd-journal/journal-send.c:88:25: error: implicit declaration of function ‘gettid’; did you mean ‘getgid’? [-Werror=implicit-function-declaration]
if (getpid() != gettid())
^~~~~~
getgid
../src/libsystemd/sd-journal/journal-send.c:88:25: warning: nested extern declaration of ‘gettid’ [-Wnested-externs]
cc1: some warnings being treated as errors
Follow-up to 50b35193ec.
Typically, in reasonably complex programs we want to realease various
caches (such as glibc allocation caches) in case of memory pressure.
Let's add explicit infrastructure for that to sd-event, that can hook
Linux' Pressure Stall Information (PSI) logic with our event loop.
This adds sd_event_add_memory_pressure() as easy, one-step API to
install an even source that is called under memory pressure.
The parameters which file to watch (the per-cgroup PSI file, or the
system-wide file /proc/pressure/memory) can be configured via env vars.
The idea is that the service manager sooner or later gains controls for
setting this up correctly.
Alternatively to the PSI a similar logic is supported but instead of
waiting for POLLPRI on a procfs/cgroupfs fd we'll wait for POLLIN on
FIFO or AF_UNIX sockets. This is useful for testing, and possibly in
other environments, for example to hook up this protocol directly with
GNOME's low memory monitor.
By default this watches on the cgroup-local PSI so that we aren't
affected by pressure on cgroups we are not related to.
I left this one as a separate commit because it is more involved.
We want people to compile with valgrind support, but we don't want to
use a slow hash function unless we're actually running under valgrind.
So the compile-time check is changed to a runtime check. When compiled
with optimization, the compiler should elide the checks on the constants,
and only leave the check for RUNNING_ON_VALGRIND. It is wrapped with
_unlikely_ so that the else branch is put in the hot path.
Most of the support for valgrind was under HAVE_VALGRIND_VALGRIND_H, i.e. we
would enable if the valgrind headers were found. The operations then we be
conditionalized on RUNNING_UNDER_VALGRIND.
But in a few places we had code which was conditionalized on VALGRIND, i.e. the
config option. I noticed because I compiled with -Dvalgrind=true on a machine
that didn't have valgrind.h, and the build failed because
RUNNING_UNDER_VALGRIND was not defined. My first idea was to add a check that
the header is present if the option is set, but it seems better to just remove
the option. The code to support valgrind is trivial, and if we're
!RUNNING_UNDER_VALGRIND, it has negligible cost. And the case of running under
valgrind is always some special testing/debugging mode, so we should just do
those extra steps to make valgrind output cleaner. Removing the option makes
things simpler and we don't have to think if something should be covered by the
one or the other configuration bit.
I had a vague recollection that in some places we used -Dvalgrind=true not
for valgrind support, but to enable additional cleanup under other sanitizers.
But that code would fail to build without the valgrind headers anyway, so
I'm not sure if that was still used. If there are uses like that, we can
extend the condition for cleanup_pools().
It's only used from sd-journal.c, and we soon would like to pass in an
sd_journal object, hence let's move this over.
This only moves code, doesn't change behaviour
Although this slightly more verbose it makes it much easier to reason
about. The code that produces the tests heavily benefits from this.
Test lists are also now sorted by test name.
A lot of tests can be defined by just their filename. Moving into their
own list keeps things simpler, especially with the next commit. It also
makes it easier to keep the lists sorted.
We nowadays check for ordering anyway at time of writing entries, hence
we don't have to do that at moment of opening, too.
Benefit of dropping this check: we can safely archive files from the
future instead of marking them as broken.
We allow reading them, and we allow creating them, but we so far did not
allow opening existing ones for write – if the machine ID is not
initialized.
Let's fix that.
(This is just to fix an asymmetry. I have no immediate use for this. But
test code should in theory be able to use this, if it runs in an
incompletely initialized environment.)
Previously, if we ran in an environment where /etc/machine-id was
not defined, we'd never bother to write it ever again. So it would stay
at all zeroes till the end of times.
Let's make this more robust: whenever we try to append an entry, let's
try to refresh it from the status quo if not initialized yet. Moreover,
when copying records from a different journal file, let's propagate the
machine ID from there.
This should make things more robust and systematic, and match how we
propagate the boot ID and the seqnum ID to some level.
This doesn't actually change much, but makes the code less surprising.
Status quo ante:
1. Open a journal file
2. If newly created set header machine ID to zero
3. If existing and open for write check if machine ID in header matches
local one, if not, refuse.
4. if open for writing, now refresh the machine ID from the local system
Of course, step 4 is pretty much pointless for existing files, as the
check in 3 made sure it is already in order or we'd refuse operating on
it anyway. With this patch this is simplified to:
1. Open a journal file
2. If newly created initialized machine ID to local machine ID
3. If existing, compare machine ID in header with local one, if not
matching refuse.
Outcome is the same.
The header of the journal file contains a boot ID field that is
currently updated whenever we open the journal file. This is not ideal:
pretty often we want to archive a journal file, and need to open it for
that. Archiving a foreign journal file should not mark it as ours, it
should just change the status flag in the file header.
The boot ID in the header is aleady rewritten whenever we write a
journal entry to the file anyway, hence all this patch effectively does
is slightly "delay" when the boot ID in the header is updated: instead
of immediately on open it is updated on the first entry that is written.
Net effect: archived journal files don't all look like they were written
to on a boot newer then they actually were
And more importantly: the "tail_entry_monotonic" field suddenly becomes
useful, since we know which boot it belongs to. Generally, monotonic
timestamps without boot ID information are useless, and this fixes it.
A new (compatible) header flag marks file where the boot_id can be
understood this way. This can be used by code that wants to make use of
the "tail_entry_monotonic" field to ensure it actually can do so safely.
This also renames the structure definition in journal-def accordingly,
to indicate we now follow the stricter semantics for it.
Currently we allocate fixed-size memory for event sources: the largest
any of the event source type needs. Discrepancy in the sizes needed for
the various event sources is quite major however: it's 144 bytes on
x86_64, i.e. more than two cache lines.
hence, let's be a tiny bit more careful, and allocate exactly as much as
we need, but not more.
This does what 404803e6ca did for the
monotonic timestamp getter, but for the realtime timestamp.
It also also makes the return value optional, exactly as for the
monotonic timestamp logic.
Let's ensure that entry seqnums remain stable and monotonic across the
entire runtime of the system, even if local storage is turned off. Let's
do this by maintainer a counter file in /run/ which we mmap() and
wherein we maintain the counter from early-boot on till late shutdown.
This takes inspiration of the kernel-seqnum file we already maintain
like that that tracks which kmsg messages we already processed. In fact,
we reuse the same code for maintaining it.
This should allow the behaviour entry seqnums to be more predictable, in
particular when journal local storage is turned off. Previously, we'd
maintain the seqnum simply by always bumping it to the maximum of the
last written entry seqnum plus one, and the biggest seqnum so far
written to the journal file on disk. If we'd never write a file on disk,
or if no journal file was existing during the initrd→seqnum transition
we'd completely lose the current seqnum position during daemon restarts
(such as the one happening during the switch-root operation).
This also will cause a journal file rotation whenever we try to write to
a journal file with multiple sequence number IDs, so that we know that
from early boot trhough the entire runtime we'll have stable sequence
numbers that do not jump, and thus can be used to determine "lost"
messages.
Cristian RodríguezandYu Watanabe
Lennart Poettering
Thorsten Kukuk
Daan De Meyer
Mike Yuan
David Tardon
Frantisek Sumsal
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
Jan Janssen