Major distributions already have elfutils >= 0.190.
Let's bump the required minimum version.
Note, elfutils 0.177 was released on 2019-08-14.
See also #38608.
Now that the necessary functions from log.h have been moved to macro.h,
we can stop including log.h in macro.h. This requires modifying source
files all over the tree to include log.h instead.
read() returns ssize_t (i.e. 64bit typically). We assigned it to int
variables in some cases (i.e. 32bit typically). Let's not be so sloppy,
and not accidentally drop 32bit on the floor.
(of course, this is not an issue IRL since we'll not have allocations
above 2^32 ever we could read into, but still, let's clean this up)
This is preparation for making our Varlink API a public API. Since our
Varlink API is built on top of our JSON API we need to make that public
first (it's a nice API, but JSON APIs there are already enough, this is
purely about the Varlink angle).
I made most of the json.h APIs public, and just placed them in
sd-json.h. Sometimes I wasn't so sure however, since the underlying data
structures would have to be made public too. If in doubt I didn#t risk
it, and moved the relevant API to src/libsystemd/sd-json/json-util.h
instead (without any sd_* symbol prefixes).
This is mostly a giant search/replace patch.
When looking at how dlopen for various libs is implemented, I found that the
macros hide too much. I find it much easier to see what is going on if 'extern'
and '= NULL' are written explicitly. After all, we don't hide those for other
definitions, e.g. our style guide says that static variables should be
initialized with '= NULL'. With that change, it's much more obvious what is
a variable declaration and what is a variable initialization.
Use 'recommended' priority for the default compression library, to
indicate that it should be prioritized over the other ones, as it
will be used to compress journals/core files.
Also use 'recommended' for kmod, as systems will likely fail to boot
if it's missing from the initrd.
Use 'suggested' for everything else.
There is one dlopen'ed TPM library that has the name generated
at runtime (depending on the driver), so that cannot be added, as it
needs to be known at build time.
Also when we support multiple ABI versions list them all, as for the
same reason we cannot know which one will be used at build time.
$ dlopen-notes.py build/libsystemd.so.0.39.0 build/src/shared/libsystemd-shared-256.so
libarchive.so.13 suggested
libbpf.so.0 suggested
libbpf.so.1 suggested
libcryptsetup.so.12 suggested
libdw.so.1 suggested
libelf.so.1 suggested
libfido2.so.1 suggested
libgcrypt.so.20 suggested
libidn2.so.0 suggested
libip4tc.so.2 suggested
libkmod.so.2 recommended
liblz4.so.1 suggested
liblzma.so.5 suggested
libp11-kit.so.0 suggested
libpcre2-8.so.0 suggested
libpwquality.so.1 suggested
libqrencode.so.3 suggested
libqrencode.so.4 suggested
libtss2-esys.so.0 suggested
libtss2-mu.so.0 suggested
libtss2-rc.so.0 suggested
libzstd.so.1 recommended
Co-authored-by: Luca Boccassi <bluca@debian.org>
We have these pretty macros, let's use them everywhere (so far we mostly
used them for newer additions only).
This PR is mostly an excercise in "perl -p -i -e", but there are some
special cases:
* idn-util.c exposes a function whose prototype in the official library
headers is marked with the "const" attribute, and this apparently does
not propagate along typeof() correctly and then
__builtin_types_compatible_p() fails later because it detects that
prototype and original function don't match in prototype.
* libbpf removed some symbols in newer versions, hence we need to define
some prototypes manually to still be able to build.
* libcryptsetup marked a symbol as deprecated we want to use (knowing it
is deprecated). By using the macros this is detected by the compiler.
We work around it via the usual warning off macros.
Note by using these macros we assume that all symbols are known during
build time. Which might not be the case. We might need to revert this
commit for some symbols if this trips up builds on older distros.
ADD_SAFE/SUB_SAFE/MUL_SAFE do addition/subtraction/multiplication
respectively with an overflow check. If an overflow occurs these return
false, otherwise true. Example: (c = a + b) would become ADD_SAFE(&c, a,
b)
INC_SAFE/DEC_SAFE/MUL_ASSIGN_SAFE are like above but they also reassign
the first argument. Example: (a += b) would become INC_SAFE(&a, b)
We use it for more than just pipe() arrays. For example also for
socketpair(). Hence let's give it a generic name.
Also add EBADF_TRIPLET to mirror this for things like
stdin/stdout/stderr arrays, which we use a bunch of times.
We usually check return value of syscalls or glibc functions by it is
negative or not, something like that `if (stat(path, &st) < 0)`.
Let's also use the same style for lseek() and friends even the type of
their return value is off_t.
Note, fseeko() returns int, instead of off_t.
json_append() is a useful wrapper around json_variant_merge(). However,
I think the naming sould be cleaned up a bit of both functions.
I thinker "merge" is the better word than "append", since it does
decidedly more than just append: it replaces existing fields of the same
name, hence "merge" sounds more appropriate. This is as opposed to the
similar operations for arrays, where no such override logic is applied
and we really just append, hence those functions are called "append"
already.
To make clearer that "merge" is about objects, and "append" about
arrays, also include "object" in the name.
Also, include "json_variant" in the name, like we do for almost all
other functions in the JSON API that take a JSON object as primary
input, and hence are kinda object methods.
Finally, let's follow the logic that helpers that combine json_build()
with some other operation get suffixed with "b" like we already have in
some cases.
Hence:
json_variant_merge() → json_variant_merge_object()
json_append() → json_variant_merge_objectb()
This mirrors nicely the existing:
json_variant_append_array()
json_vairant_append_arrayb()
This also drops the variant of json_append() that takes a va_arg
parameter (i.e. json_appendv()). We have no user of that so far, and
given the nature as a helper function only I don#t see that happening,
and if it happens after all it's trivial to bring back.
Indeed when iterating over all the PT_LOAD segment of the core dump
while trying to look for the elf headers of a given module, we iterate
over them all and try to use the first one for which we can parse a
package metadata, but the start address is never taken into account,
so absolutely nothing guarantees we actually parse the right ELF header
of the right module we are currently iterating on.
This was tested like this:
- Create a core dump using sleep on a fedora 37 container, with an
explicit LD_PRELOAD of a library having a valid package metadata:
podman run -t -i --rm -v $(pwd):$(pwd) -w $(pwd) fedora:37 bash -x -c \
'LD_PRELOAD=libreadline.so.8 sleep 1000 & SLEEP_PID="$!" && sleep 1 && kill -11 "${SLEEP_PID}" && mv "core.${SLEEP_PID}" the-core'
- Then from a fedora 38 container with systemd installed, the resulting
core dump has been passed to systemd-coredump with and without this
patch. Without this patch, we get:
Module /usr/bin/sleep from rpm bash-5.2.15-3.fc38.x86_64
Module /usr/lib64/libtinfo.so.6.3 from rpm coreutils-9.1-8.fc37.x86_64
Module /usr/lib64/libc.so.6 from rpm coreutils-9.1-8.fc37.x86_64
Module /usr/lib64/libreadline.so.8.2 from rpm coreutils-9.1-8.fc37.x86_64
Module /usr/lib64/ld-linux-x86-64.so.2 from rpm coreutils-9.1-8.fc37.x86_64
While with this patch we get:
Module /usr/bin/sleep from rpm bash-5.2.15-3.fc38.x86_64
Module /usr/lib64/libtinfo.so.6.3 from rpm ncurses-6.3-5.20220501.fc37.x86_64
Module /usr/lib64/libreadline.so.8.2 from rpm readline-8.2-2.fc37.x86_64
So the parsed package metadata reported by systemd-coredump when the module
files are not found on the host (ie the case of crash inside a container) are
now correct. The inconsistency of the first module in the above example
(sleep is indeed not provided by the bash package) can be ignored as it
is a consequence of how this was tested.
In addition to this, this also fixes the performance issue of
systemd-coredump in case of the crashing process uses a large number of
shared libraries and having no package metadata, as reported in
https://sourceware.org/pipermail/elfutils-devel/2023q2/006225.html.
When closing the FILE handle attached to a memstream, it may attempt to
do a realloc() that may fail during OOM situations, in which case we are
left with the buffer pointer pointing to NULL and buffer size > 0. For
example:
```
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
void *realloc(void *ptr, size_t size) {
return NULL;
}
int main(int argc, char *argv[])
{
FILE *f;
char *buf;
size_t sz = 0;
f = open_memstream(&buf, &sz);
if (!f)
return -ENOMEM;
fputs("Hello", f);
fflush(f);
printf("buf: 0x%lx, sz: %lu, errno: %d\n",
(unsigned long) buf, sz, errno);
fclose(f);
printf("buf: 0x%lx, sz: %lu, errno: %d\n",
(unsigned long) buf, sz, errno);
return 0;
}
```
```
$ gcc -o main main.c
$ ./main
buf: 0x74d4a0, sz: 5, errno: 0
buf: 0x0, sz: 5, errno: 0
```
This might do unexpected things if the underlying code expects a valid
pointer to the memstream buffer after closing the handle.
Found by Nallocfuzz.
In both cases, the json string is short, so we can print it, which is useful
for diagnosing invalid data in packages. But we need escape non-printable
characters.
https://bugzilla.redhat.com/show_bug.cgi?id=2152685
I went over the rest of the codebase, and it seems that other calls to
json_parse() don't have this problem.
We would deadlock when passing the data back from the forked-off process that
was doing backtrace generation back to the coredump parent. This is because we
fork the child and wait for it to exit. The child tries to write too much data
to the output pipe, and and after the first 64k blocks on the parent because
the pipe is full. The bug surfaced in Fedora because of a combination of four
factors:
- 87707784c7 was backported to v251.5, which
allowed coredump processing to be successful.
- 1a0281a3eb was NOT backported, so the output
was very verbose.
- Fedora has the ELF package metadata available, so a lot of output can be
generated. Most other distros just don't have the information.
- gnome-calendar crashes and has a bazillion modules and 69596 bytes of output
are generated for it.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2135778.
The code is changed to try to write data opportunistically. If we get partial
information, that is still logged. In is generally better to log partial
backtrace information than nothing at all.
Add dlopen_dw(), dlopen_elf() and dlopen_pcre2() to the dlopen test.
To enable adding dlopen_pcre2(), we move pcre2-dlopen.h/c from
src/journal to src/shared.
Mike Yuan
Daan De Meyer
Yu Watanabe
Jelle van der Waa
Zbigniew Jędrzejewski-Szmek
Ivan Kruglov
Michal Sekletar
Lennart Poettering
Adrian Vovk
Frantisek Sumsal
Romain Geissler
David Tardon