The overflow check for ref counting should not be subject to NDEBUG,
hence upgrade assert() → assert_se(). (The check for zero is an
immediate bug in our code, and should be impossible to trigger, hence
it's fine if the check is optimized away if people are crazy enough to
set NDEBUG, so that can stay assert())
https://github.com/systemd/systemd/pull/23099#discussion_r854341850
To make sure we don't miss any _exit() calls let's move the
coverage-related tweaks into a separate header file and include it
explicitly on the compiler command line using -include when a coverage
build is requested.
Follow-up to c6552ad381.
So far DECIMAL_STR_MAX() overestimated the types in two ways: it would
also adds space for a "-" for unsigned types.
And it would always return the same size for 64bit values regardless of
signedness, even though the longest maximum numbers for signed and
unsigned differ in length by one digit. i.e. 2^64-1 (i.e. UINT64_MAX) is
one decimal digit longer than -2^63 (INT64_MIN) - for the other integer
widths the number of digits in the "longest" decimal value is always the
same, regardless of signedness. by example: strlen("65535") ==
strlen("32768") (i.e. the relevant 16 bit limits) holds — and similar
for 8bit and 32bit integer width limits — but
strlen("18446744073709551615") > strlen("9223372036854775808") (i.e. the
relevant 64 bit limits).
Let's fix both misestimations.
With negative numbers we wouldn't account for the minus sign, thus
returning a string with one character too short, triggering buffer
overflows in certain situations.
<limits.h> calls this ULLONG_MAX. It's not clear to me where ULONGLONG_MAX
can be found. This seems to be just a mistake.
Fixes: c7ed718720 ('macro: handle overflow in ALIGN_TO() somewhat reasonably')
The helper call rounds up to next multiple of specified boundary. If one
passes a very large value as first argument, then there might not be a
next multiple. So far we ignored that. Let's handle this now and return
SIZE_MAX in this case, as special indicator that we reached the end.
Of course, IRL this should not happen. With this new change we at least
do something somewhat reasonable, leaving it to the caller to handle it
further.
We only need the PE header offset from the DOS header, not
its size. Previously, the section table could be cut off in the middle.
While we are at it, also modernize the remaining code.
The commit introduces a callback invoked from log_syntax_internal.
Use it from systemd-analyze to gather a list of units that contain
syntax warnings. A new command line option is added to make use of this.
The new option --recursive-errors takes in three possible modes:
1. yes - which is the default. systemd-analyze exits with an error when syntax warnings arise during verification of the
specified units or any of their dependencies.
3. no - systemd-analyze exits with an error when syntax warnings arise during verification of only the selected unit.
Analyzing and loading any dependencies will be skipped.
4. one - systemd-analyze exits with an error when syntax warnings arise during verification
of only the selected units and their direct dependencies.
Below are two service unit files that I created for the purposes of testing:
1. First, we run the commands on a unit that does not have dependencies but has a non-existing key-value setting (i.e. foo = bar).
> cat <<EOF>testcase.service
[Unit]
foo = bar
[Service]
ExecStart = echo hello
EOF
OUTPUT:
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify testcase.service
/home/maanya-goenka/systemd/testcase.service:2: Unknown key name 'foo' in section 'Unit', ignoring.
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=yes testcase.service
/home/maanya-goenka/systemd/testcase.service:2: Unknown key name 'foo' in section 'Unit', ignoring.
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=no testcase.service
/home/maanya-goenka/systemd/testcase.service:2: Unknown key name 'foo' in section 'Unit', ignoring.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=one testcase.service
/home/maanya-goenka/systemd/testcase.service:2: Unknown key name 'foo' in section 'Unit', ignoring.
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
2. Next, we run the commands on a unit that is syntactically valid but has a non-existing dependency (i.e. foo2.service)
> cat <<EOF>foobar.service
[Unit]
Requires = foo2.service
[Service]
ExecStart = echo hello
EOF
OUTPUT:
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify foobar.service
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
foobar.service: Failed to create foobar.service/start: Unit foo2.service not found.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=yes foobar.service
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
foobar.service: Failed to create foobar.service/start: Unit foo2.service not found.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=no foobar.service
maanya-goenka@debian:~/systemd (log-error)$ echo $?
0
maanya-goenka@debian:~/systemd (log-error)$ sudo build/systemd-analyze verify --recursive-errors=one foobar.service
/usr/lib/systemd/system/plymouth-start.service:15: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed.
/usr/lib/systemd/system/dbus.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/dbus/system_bus_socket → /run/dbus/system_bus_socket; please update the unit file accordingly.
/usr/lib/systemd/system/gdm.service:30: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
foobar.service: Failed to create foobar.service/start: Unit foo2.service not found.
maanya-goenka@debian:~/systemd (log-error)$ echo $?
1
In general we almost never hit those asserts in production code, so users see
them very rarely, if ever. But either way, we just need something that users
can pass to the developers.
We have quite a few of those asserts, and some have fairly nice messages, but
many are like "WTF?" or "???" or "unexpected something". The error that is
printed includes the file location, and function name. In almost all functions
there's at most one assert, so the function name alone is enough to identify
the failure for a developer. So we don't get much extra from the message, and
we might just as well drop them.
Dropping them makes our code a tiny bit smaller, and most importantly, improves
development experience by making it easy to insert such an assert in the code
without thinking how to phrase the argument.
Now that CONST_MAX() is a bit more foregiving, let's stick to the native
return type of sizeof() everywhere, which is size_t, instead of casting
to "unsigned", so that on the common archs we don't unnecessarily lose
the upper 32bits.
This semi-reverts d3e4029457.
When those two macros were used together in CONST_MAX(), gcc would complain
about a type mismatch. So either DECIMAL_STR_MAX() should be made size_t like
STRLEN(), or STRLEN() be made unsigned.
Since those macros are only usable on arguments of (small) fixed size, any type
should be fine (even char would work…). For buffer size specifications, both
size_t and unsigned are OK. But unsigned was used for DECIMAL_STR_MAX macros
and FORMAT_foo_MAX macros, making STRLEN the only exception, so let's adjust
STRLEN() to be unsigned too.
Also: I don't think this is currently used anywhere, but if any of those macros
were used as an argument to sprintf, size_t would require a cast. ("%*s"
requires an int/unsigned argument.)
Add SBAT support, when -Dsbat-distro value is specified. One can use
-Dsbat-distro=auto for autodetection of all sbat options. Many meson configure
options added to customize SBAT CSV values, but sensible defaults are auto
detected by default. SBAT support is required if shim v15+ is used to load
systemd-boot binary or kernel.efi (Type II BootLoaderSpec).
Fixes#19247
If the cleanup function returns the appropriate type, use that to reset the
variable. For other functions (usually the foreign ones which return void), add
an explicit value to reset to.
This causes a bit of code churn, but I think it might be worth it. In a
following patch static destructors will be called from a fuzzer, and this
change allows them to be called multiple times. But I think such a change might
help with detecting unitialized code reuse too. We hit various bugs like this,
and things are more obvious when a pointer has been set to NULL.
I was worried whether this change increases text size, but it doesn't seem to:
-Dbuildtype=debug:
before "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494520 Feb 16 15:06 build/systemd*
after "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494576 Feb 16 15:10 build/systemd*
now:
-rwxrwxr-x 1 zbyszek zbyszek 4117672 Feb 16 14:36 build/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 4494640 Feb 16 15:15 build/systemd*
-Dbuildtype=release:
before "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:09 build-rawhide/systemd*
after "tree-wide: return NULL from freeing functions":
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:10 build-rawhide/systemd*
now:
-rwxrwxr-x 1 zbyszek zbyszek 5252256 Feb 14 14:47 build-rawhide/libsystemd.so.0.30.0*
-rwxrwxr-x 1 zbyszek zbyszek 1834184 Feb 16 15:16 build-rawhide/systemd*
I would expect that the compiler would be able to elide the setting of a
variable if the variable is never used again. And this seems to be the case:
in optimized builds there is no change in size whatsoever. And the change in
size in unoptimized build is negligible.
Something strange is happening with size of libsystemd: it's bigger in
optimized builds. Something to figure out, but unrelated to this patch.
sd-boot has a copy of a subset of codes from libbasic. This makes
sd-boot share the code with libbasic, and dedup the code.
Note, startswith_no_case() is dropped from sd-boot, as
- it is not used,
- the previous implementation is not correct,
- gnu-efi does not have StrniCmp() or so.
The idea is that we have strvs like list of server names or addresses, where
the majority of strings is rather short, but some are long and there can
potentially be many strings. So formattting them either all on one line or all
in separate lines leads to output that is either hard to read or uses way too
many rows. We want to wrap them, but relying on the pager to do the wrapping is
not nice. Normal text has a lot of redundancy, so when the pager wraps a line
in the middle of a word the read can understand what is going on without any
trouble. But for a high-density zero-redundancy text like an IP address it is
much nicer to wrap between words. This also makes c&p easier.
This adds a variant of TABLE_STRV which is wrapped on output (with line breaks
inserted between different strv entries).
The change table_print() is quite ugly. A second pass is added to re-calculate
column widths. Since column size is now "soft", i.e. it can adjust based on
available columns, we need to two passes:
- first we figure out how much space we want
- in the second pass we figure out what the actual wrapped columns
widths will be.
To avoid unnessary work, the second pass is only done when we actually have
wrappable fields.
A test is added in test-format-table.
We would print the whole string as a single super-long line. Let's nicely
break the text into lines that fit on the screen.
$ COLUMNS=70 build/resolvectl --no-pager nta
Global: home local intranet 23.172.in-addr.arpa lan
18.172.in-addr.arpa 16.172.in-addr.arpa 19.172.in-addr.arpa
25.172.in-addr.arpa 21.172.in-addr.arpa d.f.ip6.arpa
20.172.in-addr.arpa 30.172.in-addr.arpa 17.172.in-addr.arpa
internal 168.192.in-addr.arpa 28.172.in-addr.arpa
22.172.in-addr.arpa 24.172.in-addr.arpa 26.172.in-addr.arpa
corp 10.in-addr.arpa private 29.172.in-addr.arpa test
27.172.in-addr.arpa 31.172.in-addr.arpa
Link 2 (hub0):
Link 4 (enp0s31f6):
Link 5 (wlp4s0):
Link 7 (virbr0): adsfasdfasdfasd.com 21.172.in-addr.arpa lan j b
a.com home d.f.ip6.arpa b.com local 16.172.in-addr.arpa
19.172.in-addr.arpa 18.172.in-addr.arpa 25.172.in-addr.arpa
20.172.in-addr.arpa k i h 23.172.in-addr.arpa
168.192.in-addr.arpa d g intranet 17.172.in-addr.arpa c e.com
30.172.in-addr.arpa a f d.com e internal
Link 8 (virbr0-nic):
Link 9 (vnet0):
Link 10 (vb-rawhide):
Link 15 (wwp0s20f0u2i12):
Latest glibc has deprecated mallinfo(), so it might become unavailable at some point
in the future. There is malloc_info(), but it returns XML, ffs. I think the information
that we get from mallinfo() is quite useful, so let's use mallinfo() if available, and
not otherwise.
I had to move STRV_MAKE to macro.h. There is a circular dependency between
extract-word.h, strv.h, and string-util.h that makes it hard to define the
inline function otherwise.
When accessing journal files we generally are fine when values change
beneath our feet, while we are looking at them, as long as they change
from something valid to zero. This is required since we nowadays
forcibly unallocate journal files on vacuuming, to ensure they are
actually released.
However, we need to make sure that the validity checks we enforce are
done on suitable copies of the fields in the file. Thus provide a macro
that forces a copy, and disallows the compiler from merging our copy
with the actually memory where it is from.
This restores proper speed with asan builds with gcc 9.1.1.
Fixes#12997.
$ rpm -q gcc
gcc-9.1.1-2.fc31.x86_64
$ time ASAN_OPTIONS=strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1 UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1 build-rawhide-sanitize/test-conf-parser
(old) 86.99s user 20.22s system 361% cpu 29.635 total
(new) 3.05s user 0.29s system 99% cpu 3.377 total
Size is increased a bit:
$ size build/systemd
(old) 1683421 246100 1208 1930729 1d75e9 build/systemd
(new) 1688237 246100 1208 1935545 1d88b9 build/systemd
... but that's <0.1%, so we don't really care.
Lennart Poettering
Yu Watanabe
Frantisek Sumsal
Jan Janssen
Thomas Haller
Maanya Goenka
Zbigniew Jędrzejewski-Szmek
Dimitri John Ledkov
Arian van Putten
Marc-André Lureau