Observerd on Ubuntu 24.04 with GCC 13 on arm64 architecture.
```
../src/shared/journal-importer.c: In function ‘journal_importer_process_data’:
../src/shared/journal-importer.c:344:30: error: ‘line’ may be used uninitialized [-Werror=maybe-uninitialized]
344 | if (!journal_field_valid(line, n - 1, true)) {
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/shared/journal-importer.c:295:23: note: ‘line’ was declared here
295 | char *line, *sep;
| ^~~~
cc1: all warnings being treated as errors
```
There are already varlink methods for `ResolveHostname`,
`ResolveAddress`, and `ResolveRecord`. Use those in `resolvectl` instead
of the dbus equivalents.
Follow-up to #42422:
- Rename process_image() to context_process_image(), since it now
operates on a Context object.
- Use IN_SET() in image_type_can_sysupdate() instead of a switch.
- Name the return parameters of context_list_components() ret_xyz, per
our coding style.
- Drop a redundant "else" after a return in vl_method_check_new().
This puts the scaffolding in place for a varlink interface, but so far
it only adds a `io.systemd.Sysupdate.CheckNew()` method. Varlinkifying
the other verbs on `systemd-sysupdate` will happen in follow-up PRs, but
I thought I’d try and land this one early to:
* Get review of the overall varlinkification scaffolding
* Lower the chance of big merge conflicts with others’ work by getting
the more invasive changes out of the way
* Get the scaffolding in place so others can start to build on it if
they wish (although I am currently working on porting the other existing
verbs)
It rearranges how the `Context` struct is allocated so that it’ll be
easier to add per-method/verb context structs which contain it in
future. It also changes all the `sysupdate.c` code to use arguments from
`Context` rather than `arg_*` globals, allowing them to be specified as
varlink parameters in future.
It also moves the existing `systemd-sysupdate.{timer,service}` units
(which periodically run `systemd-sysupdate update`) to
`systemd-sysupdate-update.{timer,service}` to clear space for a
`systemd-sysupdate@.service` and `systemd-sysupdate.socket` to act as a
varlink entry point.
/cc @AdrianVovk
postmarketOS image can be built with for example the following mkosi.local.conf:
```
[Distribution]
Distribution=postmarketos
[Output]
Format=disk
[Build]
UseSubvolumes=yes
[Runtime]
Firmware=uefi
```
This is preparation for using varlink methods more in resolvectl. In
particular, this is helpful for proving more accurate error messages,
and maintaining compatibility with existing error messages from the
DBus API.
Generalize json_dispatch_address from nss-resolve, and add support for
strings to be compatible with the existing json_dispatch_{in,in6}_addr
helpers. This will be used in a later commit.
There are several places where address family is dispatched from JSON,
so take json_dispatch_address_family from networkd and put it in
json-util.c.
Update the all instances in the tree to use this new function, adding
SD_JSON_RELAX to the dispatch flags if AF_UNSPEC should be allowed
in that case.
This is the first varlink method added to sysupdate. The D-Bus interface
(via sysupdated) will remain for now; the varlink interface will exist
in parallel.
This method can be called via:
```
varlinkctl call ./path/to/systemd-sysupdate \
io.systemd.SysUpdate.CheckNew \
'{"target":{"class":"host"}}'
SYSTEMD_SYSUPDATE_NO_VERIFY=1 \
varlinkctl call ./path/to/systemd-sysupdate \
io.systemd.SysUpdate.CheckNew \
'{"target":{"class":"component","name":"some-component"}}'
```
This includes some changes to run the integration tests again using the
varlink interface rather than running `systemd-sysupdate` directly, to
test the new interface.
This adds the scaffolding for being able to call sysupdate via varlink,
but it doesn’t yet define or implement any methods. Those will come in
following commits.
The existing `systemd-sysupdate.service` and `systemd-sysupdate.timer`
(which periodically ran `systemd-sysupdate update`) have been renamed
to `systemd-sysupdate-update.{service,timer}` to make way for a new
`systemd-sysupdate@.service` and `systemd-sysupdate.socket` file to
handle varlink activation.
Compatibility symlinks have been added for them.
If a varlink method takes a struct/object as a parameter, and it’s
marked as `SD_JSON_MANDATORY`, and it has an optional field inside it
which is *not* marked as `SD_JSON_MANDATORY`, we want to not require
that field to be set.
Previously, due to using the `merged_flags` from both the mandatory
struct and the optional field, `SD_JSON_MANDATORY` was effectively
always set on the optional field even if we didn’t want it. This
resulted in an error being emitted if the mandatory struct was provided
in a varlink call, but without the optional field.
Fix that by validating the field’s presence only against its own flags
and not also the flags of its parent.
Adds a unit test to prevent regressions.
This will be used in upcoming commits to varlinkify `systemd-sysupdate`;
it will need a way to identify targets over varlink, and the existing
way with a `Target` over D-Bus seems to work quite well.
`process_image()` has historically used `umount_and_freep` to clean up
the mounted directory locally, but callers to it have used
`umount_and_rmdir_and_freep`.
No directory is created after any of the error return paths in
`process_image()`, so it should probably be using
`umount_and_rmdir_and_freep` too.
This is another step towards varlinkifying the program, as it means the
various verb implementations are no longer relying on global state from
the command line.
As part of this, move init of the `Context` struct into a new
`context_from_cmdline()` function.
Additionally pass some context into config parsing `userdata` arguments,
as various config parsers were using `arg_root` via a sneaky `extern`.
This introduces no functional changes.
There’s no need for it to be heap allocated — there’s only ever one
instance of it, and it’s allocated for the lifetime of a `verb_*()`
function.
Simplify things a bit by making it stack allocated. This will also help
with upcoming commits where we introduce derived context structs to help
with varlinkifying sysupdate. By allowing `Context` to be stack
allocated we can include it in the derived context structs.
As part of this, rename `context_make_{offline,online}()` to
`context_load_{offline,online}()` for clarity (since they no longer init
the struct).
This introduces no functional changes.
`process_image()` is always called immediately before (almost) every
`context_make_online()` or `context_make_offline()`, and the structures
it allocates have the same lifetime as `Context`, so we might as well
factor them all together to reduce duplication.
This will also simplify the following commit, which changes heap
allocation of `Context`s, and simplify upcoming changes to factor out
`arg_*` handling.
The call in `verb_pending_or_reboot()` is safe because it already
validates that `arg_image` is `NULL`, hence `process_image()` will bail
out early.
This introduces no functional changes.
This makes it like all the other verbs and therefore easier to refactor.
At the same time, remove the separate `component` argument and instead
use the `component` set on the `Context`. This guards against bugs, as
various parts of the `Context` state depend on the component (for
example, `installdb_fd`) and overriding the component without also
overriding its dependent variables will lead to bugs.
If the FD has already been opened, return 1 as if opening was
successful, rather than returning 0 as if it gave `ENOENT`.
This fixes doing multiple installdb operations on a single `Context`.
Signed-off-by: Philip Withnall <pwithnall@gnome.org>
The example states that the first /bin/echo invocation (using ${ONE})
receives the argument 'one' (with literal single quotes). However,
Environment=ONE='one' strips the syntactic single quotes during
unquoting — see systemd.syntax(7), "Quotes themselves are removed" —
so ONE holds the value one, and ${ONE} (exact-value substitution,
always a single argument) yields the argument one without quotes.
Fixes#42442
Signed-off-by: Wang Yu <wangyu@uniontech.com>
Previously the key was unknown so add the correct mapping as it does not follow the general
case for MSI Laptops.
[ 192.562000] atkbd serio0: Unknown key released (translated set 2, code 0xd7 on isa0060/serio0).
[ 192.562011] atkbd serio0: Use 'setkeycodes e057 <keycode>' to make it known.
Add it currently as a definition specific for this model but can be generalized to other MSI
Laptops if this issue is present also elsewhere.
When running in a mkosi namespaced env tmpfiles fails to set ACLs:
Running create action for entry a /buildroot/var/log/journal
Setting access ACL u::rwx,g::r-x,g:adm:r-x,m::r-x,o::r-x on /buildroot/var/log/journal
Setting access ACL "u::rwx,g::r-x,g:adm:r-x,m::r-x,o::r-x" on /buildroot/var/log/journal failed: Invalid argument
If EINVAL is returned and we are in a chroot, skip gracefully via
EOPNOTSUPP. The ACLs will be set on first boot.
event_log_record_extract_firmware_description() walks the device path
of a UEFI_IMAGE_LOAD_EVENT taken from the firmware TPM2 measurement log.
The per-node loop checks the remaining bytes against the node and its
declared length, but never that dp->length covers the 4-byte node header
offsetof(packed_EFI_DEVICE_PATH, path).
For a Media/File-Path node with length 3, the file-name extraction
computes dp->length - offsetof(packed_EFI_DEVICE_PATH, path) == 3 - 4,
which wraps to SIZE_MAX. utf16_to_utf8() treats SIZE_MAX as unbounded
and runs char16_strlen() over dp->path, reading past the log buffer; a
length of 0 also leaves dp non-advancing.
efi_get_boot_option() in src/shared/efi-api.c already rejects such nodes
with "if (dpath->length < 4) break;"; do the same here.
Re-enables `--set-credential=` / `--load-credential=` under
`--coco=sev-snp` by packaging credentials into a cpio appended to the
initrd, mirroring what `systemd-stub` does for ESP-sourced credentials.
The initrd is covered by the launch measurement via `kernel-hashes=on`,
so the credentials are too.
Tested end-to-end on an SNP-capable host: credentials passed via
`--set-credential=` land in `/run/credentials/@encrypted/` inside the
guest.
Use sd_json_variant_is_blank_array() instead of is_blank_object() for
p.addresses and p.names, which are declared as JSON arrays. The wrong
predicate never triggered, allowing empty arrays to bypass the guards:
for p.names this caused a size_t underflow leading to an out-of-bounds
heap write; for p.addresses it returned success with no addresses.
Add explicit n_addresses == 0 guards after the family-filter loops so
entries with unsupported families also return NOTFOUND rather than
crashing on a NULL dereference.
In gethostbyname3_r (family-specific entry point), return NO_DATA for
all zero-address results — both blank array and all-filtered — since
both mean "name resolved, no record of the requested family". Keep
HOST_NOT_FOUND in gethostbyname4_r (both-families) where a blank or
all-unsupported result genuinely means the name was not found.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
In Meta production we have been considering using journald more widely
for some time. One of the blockers to doing that which I have noticed is
that often journald seems to have vastly less data after lockups/power
failures compared to plain files, which is not great when debugging
outages.
On small write rates this tends to be hard to reproduce, but when
writing thousands of messages a second, an unclean shutdown can result
in the end result being an active journal file with a header that
records an arena larger than the data that actually reached disk. What
happens is then that journalctl then discards the entire file(!),
completely ignoring that there is a huge amount of data which is
actually perfectly readable.
The reason for that is that the journal header is updated on every
append, while the file size and newly written arena contents are only
made durable on the filesystem's own schedule. After a crash, the header
can therefore describe writes which were logically completed by journald
but whose backing data or file metadata never reached disk.
Take the following example of how this can happen at high log rates:
1. journald appends objects into an mmap()ed arena, periodically growing
the file with fallocate() in FILE_SIZE_INCREASE (8M) steps and advancing
the header's arena_size tail pointers as it goes along.
2. The header is dirtied on every append, and its arena_size is advanced
at each fallocate(). It is, from the kernel's perspective, an ordinary
data page and is only made durable by the kernel's periodic page cache
writeback on its own schedule. The file's length, by contrast, is
metadata, made durable only when the filesystem commits a transaction
(or on an fsync(), which journald does not issue between sync
intervals).
3. journald marks journals NOCOW, so the header's data block is
overwritten in place and is decoupled from the size metadata. Nothing
orders the two with respect to each other. Writeback therefore can
routinely persist a header whose arena_size has run ahead of the file
length recorded on disk.
4. Power is lost. On the next boot the persisted header reflects an
arena_size and tail pointers which have been advanced for appends.
However their payload and the file metadata were never committed, so
header_size + arena_size now points well past the end of the file as it
exists on disk.
5. journal_file_verify_header() then rejects this with -ENODATA:
if (... || header_size + arena_size > (uint64_t) f->last_stat.st_size)
return -ENODATA;
That is correct when opening for writing, because we must not append to
a file whose recorded state we cannot trust, and the caller must rotate
it away. But the same check also runs on read only opens, where it is
actively harmful. In the case of journalctl, the entire file is skipped,
even though the data hash table, the field hash table, and the head of
the array all are present and fully intact, and the great majority of
entries are physically present. In fact, only a very small part of the
most recently written tail is missing, but everything before is
readable. This results in mistakenly rejecting the entire file as
corrupt.
This happens extremely frequently on machines with high write rates
during power cuts or lockups. In testing writing ~7500 msg/s through
journald and then cutting power, I reproduced it in ten out of ten
attempts across different machines.
In each case, the header was left claiming ~296M of arena while only
~192-208M had reached disk. In this case, journalctl reports that it has
recovered 0 of ~335000 messages. Whether a given crash trips the
condition depends on where it falls relative to the header's writeback,
but when it does, the loss today is total. After this patch the vast
majority of messages can be retrieved.
Let's fix this by keeping the rejection for writing, but for read-only
opens, let's just clamp the arena to the real file size and skip the
consistency checks on the now unreliable tail pointers. The reader will
walk the entry array chain from its intact head and stop at the
truncation point by the bounds check that already exists, so there's no
need to do any more than that there.
`systemd-analyze security --user foo.service` currently flags units
without `User=` as running as root. For user manager instances this is
impossible: per systemd.exec(5), switching user identity is not
permitted there, so the service always runs under the calling user's
UID.
Track the runtime scope inside SecurityInfo and short-circuit
security_info_runs_privileged() and assess_user() for
RUNTIME_SCOPE_USER, so that User=/DynamicUser=, SupplementaryGroups=
and RemoveIPC= are no longer marked as if the service ran as root in
both the bus-backed and --offline paths.
Fixes#40292
Signed-off-by: Shihao Ren <renshihao.rsh@bytedance.com>
unit_name_mangle_with_suffix() is quite benevolent by default and allows
the unit to "transition" into a different unit type than what's
requested via its suffix argument. For example, calling
unit_name_mangle_with_suffix() with "/foo/bar" as a unit name and
".service" as a suffix would give you "foo-bar.mount", without any
warning or error.
This could then lead to a quite confusing errors in certain situations:
```
~# systemd-run --remain-after-exit --unit /foo/bar true
Failed to start transient service unit: Cannot set property RemainAfterExit, or unknown property.
```
Given we can't change the default behaviour of
unit_name_mangle_with_suffix() as some parts of systemd already depend
on its "benevolence" (like systemctl), let's introduce a new flag -
UNIT_NAME_MANGLE_STRICT - that checks if the mangled/resolved unit
name's suffix matches the requested one and errors out if not.
With the flag used throughout systemd-run's code, the error in the above
case is now a bit more clear:
```
~# build/systemd-run --remain-after-exit --unit /foo/bar true
Path "/foo/bar" resolves to unit type "mount", but "service" is expected as unit.
Failed to mangle unit name: Invalid argument
```
Resolves: #39996
In home_unlocking_finish(), the success path calls operation_result_unref()
with the local variable r and the uninitialized error object. If either
user_record_good_authentication() or home_save_record() fails (both are
logged as "ignoring"), r is left negative and the D-Bus caller receives
an error reply despite the home having been unlocked successfully.
This causes PAM to reject the session even though the home directory is
mounted and accessible.
Fix by passing 0 and NULL — consistent with every other success path in
the file (home_locking_finish(), home_activation_finish(), etc.).
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
The data and field hash table chains have the same problem the previous
commit fixed for entry array chains. New data and field objects are
linked at the tail of their hash bucket by patching the previous tail
object's next_hash_offset in place, so after a crash a persisted
predecessor (or the bucket head) can point at an object whose body never
reached disk.
journal_file_find_data_object_with_hash() and
journal_file_find_field_object_with_hash() walk those chains while
resolving matches, and on -EADDRNOTAVAIL/-EBADMSG from
journal_file_move_to_object() they simply return the error directly.
That propagates up to real_journal_next(), which discards the whole file
from the query.
Give those two lookups the same tolerance: on a read-only file, treat an
unreadable chain node as the end of the bucket chain.
Yu Watanabe
Lennart Poettering
Nick Rosbrook
dirhamtriyadi
Philip Withnall
Wang Yu
Fede2782
Luca Boccassi
Syed Mohammed Nayyar
dongshengyuan
Shihao Ren
Chris Down