The tag-match loop emits 6 instructions per tag into a fixed 512-entry
stack array without a bounds check (unlike the subsystem loop below),
so enough tags overflow the stack. Refuse with -E2BIG instead.
Follow-up for b1c097af8d
Let's not manually create comments, but use the usual
varlink_idl_format_comment() function which does line break handling
properly.
This polishes the output for very narrow outputs, since we'll line break
these synthetic comments too.
This makes each `foo_cflags` dependency define a `SYSTEMD_CFLAGS_MARKER_FOO`
macro, and checks if the macro is set when headers provided by external
libraries are included.
With this, we can fail fast at compile time if necessary `_cflags`
dependencies are omitted in meson.build. Missing dependencies found by this
mechanism have been added across the tree.
The idea is that we can build a container by building a single-binary
systemd:
```console
$ meson setup build-static --default-library=static --prefer-static --auto-features=disabled -Dbuild-static=true -Dsystemd-multicall-binary=true && ninja -C build-static systemd
$ mkdir /var/tmp/container/usr/lib -p
$ cp build-static/systemd /var/tmp/container/usr/lib/
$ echo 'ID=quick' >/var/tmp/container/usr/lib/os-release
$ systemd-nspawn --restrict-address-families=af_unix --register=no --private-users=managed -D /var/tmp/container/ /usr/lib/systemd
░ Spawning container container on /var/tmp/container.
░ Press Ctrl-] three times within 1s to kill container; two times followed by r
░ to reboot container; two times followed by p to poweroff container.
Selected user namespace base 1855193088 and range 65536.
systemd 262~devel running in system mode (-PAM -AUDIT +SELINUX -APPARMOR +IMA +IPE +SMACK -SECCOMP -GCRYPT +GNUTLS +OPENSSL -ACL +BLKID +CURL -ELFUTILS -FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP -LIBARCHIVE)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Detected first boot.
Welcome to Linux!
Initializing machine ID from container UUID.
Failed to open netlink, ignoring: Address family not supported by protocol
Applying preset policy.
Populated /etc with preset unit settings.
Unit default.target not found.
Falling back to graphical.target.
Mount unit not supported, skipping *MountsFor= dependencies.
Queued start job for default target graphical.target.
[ OK ] Reached target sysinit.target.
[ OK ] Reached target basic.target.
System is tainted: unmerged-bin:var-run-bad
[ OK ] Reached target multi-user.target.
[ OK ] Reached target graphical.target.
Startup finished in 61ms.
```
The container can be reloaded with SIGTERM, powered off with SIGRTMIN+4,
etc. SIGRTMIN+5 should cause a reboot but it currently fails:
```
...
Rebooting.
Container container is being rebooted.
Failed to attach root directory: Invalid argument
Failed to receive mount namespace fd from outer child: Input/output error
```
It's a bug … somewhere, but probably not caused by the linking changes
being done here.
Some state files were prefixed with the machine name which protected
against collisions when running concurrent machines in a systemd unit
with RuntimeDirectory= set. However, the socket files tpm.sock and
control were not, which caused a race. Prefixing these socket files
does not work because the path gets too long when both the subdirectory
and the socket file within it get unique identifiers.
Instead of prefixing all files, we can rather always create a
subdirectory and use simple names within in. This makes paths shorter
in the normal case and protects against races with RuntimeDirectory=
were instead of directly reusing RUNTIME_DIRECTORY we also create the
normal per-machine subdirectory. Since this here is about runtime state
it should not impact any running VMs/containers. The ssh-proxy looks
only at the normal case and does not support RUNTIME_DIRECTORY, so no
impact there as well.
When we try to build with -static, the linker refuses dlopen and
various functions related to nss. This commit adds the basic support
to compile with -static and allow the link to pass.
Add more shutdown timestamps, export them via D-Bus/Varlink, use them in
`systemd-analyze time`, and preserve them across kexec via LUO. This is
useful to measure reboot performance with added measurement points and
more granular intervals.
journal_file_read_tail_timestamp() is called unconditionally in
next_beyond_location() for every file on every iteration step,
resulting in O(N x files) volatile mmap reads. For large queries
like 'journalctl -n 1000000' this makes the command unusably slow
(~5 minutes on systems with many journal files).
Rather than suppressing the call entirely (which would make the
inotify path fully load-bearing for cross-boot ordering), rate-limit
it to at most once per second per file. This reduces the overhead
from O(N x files) to O(T x files) where T is the iteration time in
seconds, while still providing periodic refresh as a fallback for
any missed inotify events and keeping cross-boot ordering
reasonably fresh.
Embed a RateLimit struct in JournalFile for this purpose.
Measured improvement on a real system: 5:24 -> 2:39 (-51%) for
'journalctl -n 1000000'.
Fixes: #42521
Regression introduced in: #26355 (34af74946e)
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
After walking a hash chain, the loop exits with p == 0. The error()
call for a tail_hash_offset mismatch passed p as the file offset,
printing 0000000000000000 instead of the actual last data object.
Pass 'last' instead, which holds the offset of the final chain entry.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Add a Varlink interface for systemd-cryptenroll, building on the EnrollContext
introduced previously. A single Enroll method covers password, recovery-key and
FIDO2 enrollment; PKCS#11 and TPM2 are not exposed for now (they are not
part of the EnrollMechanism allowlist, so the generic InvalidParameter error
applies). A ListSlots method enumerates the currently enrolled keyslots.
The dispatcher populates the same EnrollContext the command line uses and then
runs the shared enroll_now()/prepare_luks()/wipe_slots() paths, so both
front-ends behave identically. FIDO2 enrollment that requires user presence
reports an imminent touch via a non-terminating "state":"touch" reply when the
caller passes 'more'. Credential material (password, FIDO2 PIN, recovery key)
is handled as sensitive, and key files may be passed either by path or as an
fd index.
The server is allocated root-only plus caller's-own-UID, with the listening
socket created in 0644 mode.
Replaces: #31096
This introduce a vtable for journal tagging feature in sd-journal,
and makes libgcrypt dependent features loaded by users (journald,
journalctl, journal-remote, and unit tests) when necessary.
Then, move several components for journal tagging in JournalFile
to JournalAuthContext.
This also introduces wrapper functions that checks gcrypt support.
There are already varlink methods for `ResolveHostname`,
`ResolveAddress`, and `ResolveRecord`. Use those in `resolvectl` instead
of the dbus equivalents.
Generalize json_dispatch_address from nss-resolve, and add support for
strings to be compatible with the existing json_dispatch_{in,in6}_addr
helpers. This will be used in a later commit.
There are several places where address family is dispatched from JSON,
so take json_dispatch_address_family from networkd and put it in
json-util.c.
Update the all instances in the tree to use this new function, adding
SD_JSON_RELAX to the dispatch flags if AF_UNSPEC should be allowed
in that case.
This adds the scaffolding for being able to call sysupdate via varlink,
but it doesn’t yet define or implement any methods. Those will come in
following commits.
The existing `systemd-sysupdate.service` and `systemd-sysupdate.timer`
(which periodically ran `systemd-sysupdate update`) have been renamed
to `systemd-sysupdate-update.{service,timer}` to make way for a new
`systemd-sysupdate@.service` and `systemd-sysupdate.socket` file to
handle varlink activation.
Compatibility symlinks have been added for them.
If a varlink method takes a struct/object as a parameter, and it’s
marked as `SD_JSON_MANDATORY`, and it has an optional field inside it
which is *not* marked as `SD_JSON_MANDATORY`, we want to not require
that field to be set.
Previously, due to using the `merged_flags` from both the mandatory
struct and the optional field, `SD_JSON_MANDATORY` was effectively
always set on the optional field even if we didn’t want it. This
resulted in an error being emitted if the mandatory struct was provided
in a varlink call, but without the optional field.
Fix that by validating the field’s presence only against its own flags
and not also the flags of its parent.
Adds a unit test to prevent regressions.
In Meta production we have been considering using journald more widely
for some time. One of the blockers to doing that which I have noticed is
that often journald seems to have vastly less data after lockups/power
failures compared to plain files, which is not great when debugging
outages.
On small write rates this tends to be hard to reproduce, but when
writing thousands of messages a second, an unclean shutdown can result
in the end result being an active journal file with a header that
records an arena larger than the data that actually reached disk. What
happens is then that journalctl then discards the entire file(!),
completely ignoring that there is a huge amount of data which is
actually perfectly readable.
The reason for that is that the journal header is updated on every
append, while the file size and newly written arena contents are only
made durable on the filesystem's own schedule. After a crash, the header
can therefore describe writes which were logically completed by journald
but whose backing data or file metadata never reached disk.
Take the following example of how this can happen at high log rates:
1. journald appends objects into an mmap()ed arena, periodically growing
the file with fallocate() in FILE_SIZE_INCREASE (8M) steps and advancing
the header's arena_size tail pointers as it goes along.
2. The header is dirtied on every append, and its arena_size is advanced
at each fallocate(). It is, from the kernel's perspective, an ordinary
data page and is only made durable by the kernel's periodic page cache
writeback on its own schedule. The file's length, by contrast, is
metadata, made durable only when the filesystem commits a transaction
(or on an fsync(), which journald does not issue between sync
intervals).
3. journald marks journals NOCOW, so the header's data block is
overwritten in place and is decoupled from the size metadata. Nothing
orders the two with respect to each other. Writeback therefore can
routinely persist a header whose arena_size has run ahead of the file
length recorded on disk.
4. Power is lost. On the next boot the persisted header reflects an
arena_size and tail pointers which have been advanced for appends.
However their payload and the file metadata were never committed, so
header_size + arena_size now points well past the end of the file as it
exists on disk.
5. journal_file_verify_header() then rejects this with -ENODATA:
if (... || header_size + arena_size > (uint64_t) f->last_stat.st_size)
return -ENODATA;
That is correct when opening for writing, because we must not append to
a file whose recorded state we cannot trust, and the caller must rotate
it away. But the same check also runs on read only opens, where it is
actively harmful. In the case of journalctl, the entire file is skipped,
even though the data hash table, the field hash table, and the head of
the array all are present and fully intact, and the great majority of
entries are physically present. In fact, only a very small part of the
most recently written tail is missing, but everything before is
readable. This results in mistakenly rejecting the entire file as
corrupt.
This happens extremely frequently on machines with high write rates
during power cuts or lockups. In testing writing ~7500 msg/s through
journald and then cutting power, I reproduced it in ten out of ten
attempts across different machines.
In each case, the header was left claiming ~296M of arena while only
~192-208M had reached disk. In this case, journalctl reports that it has
recovered 0 of ~335000 messages. Whether a given crash trips the
condition depends on where it falls relative to the header's writeback,
but when it does, the loss today is total. After this patch the vast
majority of messages can be retrieved.
Let's fix this by keeping the rejection for writing, but for read-only
opens, let's just clamp the arena to the real file size and skip the
consistency checks on the now unreliable tail pointers. The reader will
walk the entry array chain from its intact head and stop at the
truncation point by the bounds check that already exists, so there's no
need to do any more than that there.
The data and field hash table chains have the same problem the previous
commit fixed for entry array chains. New data and field objects are
linked at the tail of their hash bucket by patching the previous tail
object's next_hash_offset in place, so after a crash a persisted
predecessor (or the bucket head) can point at an object whose body never
reached disk.
journal_file_find_data_object_with_hash() and
journal_file_find_field_object_with_hash() walk those chains while
resolving matches, and on -EADDRNOTAVAIL/-EBADMSG from
journal_file_move_to_object() they simply return the error directly.
That propagates up to real_journal_next(), which discards the whole file
from the query.
Give those two lookups the same tolerance: on a read-only file, treat an
unreadable chain node as the end of the bucket chain.
generic_array_get() which is used for the unfiltered iteration path in
the previous commit treats a chain pointer that resolves past the end of
the file as the end of the chain. In that case, moving to the missing
array object returns -EADDRNOTAVAIL (or -EBADMSG), and it either stops
(going downwards) or steps back to the previous array (going upwards).
However, generic_array_bisect(), which is used for filtered or seeking
reads does not. On -EADDRNOTAVAIL/-EBADMSG from
journal_file_move_to_object(), it instead returns the error directly to
the caller, which propagates out through
sd_journal_next()/sd_journal_previous() and aborts the query.
The per-data entry array chain has the same issue as the global one,
since n_entries and entry_array_offset are (re)written in place as
entries are linked, and thus after a crash they can reference more
arrays than actually reached the disk. That is to say in practical
terms, a journal recovered for reading by the previous commit could
nevertheless still drop matching entries from `journalctl FIELD=value`,
and a seqnum or time seek into the lost region could fail outright.
Let's give generic_array_bisect() the same tolerance generic_array_get()
already has. That is, when moving to an entry array object fails, treat
the chain as ending at the previous array. This means that the result
matches what generic_array_get() would yield for the same file.
journal_file_setup_data_hash_table() allocates s * sizeof(HashItem)
bytes for the hash table but then only zeroes s bytes, leaving 15/16 of
the entries uninitialized. This corrupts the hash chain in any newly
created journal file.
The adjacent journal_file_setup_field_hash_table() already uses the
correct size.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Define a new io.systemd.SysUpdate.Notify Varlink interface with a single
OnCompletedUpdate() method, and after sysupdate successfully installs an
update, invoke that method on every socket linked into
/run/systemd/sysupdate/notify/ via varlink_execute_directory(). This
gives other components a hook to react to applied updates (e.g. recompute
a TPM policy, link a freshly downloaded kernel, refresh extensions).
The notification carries the component name, the installed version and the
list of updated resources (transfer id + on-disk path). Subscribers are
free to ignore the parameters and just treat the call as a trigger.
Setting SYSTEMD_SYSUPDATE_FORCE_NOTIFY=1 forces the notification to be sent
even when no update was applied (in which case no resource list is included),
so follow-up work can be triggered unconditionally.
Fixes: #35988
Luca Boccassi
Lennart Poettering
Yu Watanabe
Kai Lüke
Zbigniew Jędrzejewski-Szmek
dongshengyuan
Nick Rosbrook
Philip Withnall
Chris Down