Commit Graph
70869 Commits
Author SHA1 Message Date
Yu WatanabeandCosta Tsaousis cb2b0326ee sd-journal: drop to use Hashmap to manage journal files per boot ID
As reported at https://github.com/systemd/systemd/pull/30209#issuecomment-1831344431,
using hashmap in frequently called function reduces performance.
Let's replace it with a single array and bsearch.

Replaces #29366.

Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
2024-02-12 02:42:36 +09:00
Yu WatanabeandCosta Tsaousis f7f062bf9f tree-wide: set SD_JOURNAL_ASSUME_IMMUTABLE where appropriate
Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
2024-02-12 02:42:36 +09:00
Yu WatanabeandCosta Tsaousis bd1af1d5fb sd-journal: introduce SD_JOURNAL_ASSUME_IMMUTABLE flag
The flag can be used when it is not necessary to follow journal file
update.

Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
2024-02-12 02:41:28 +09:00
Yu Watanabe 51b2bcf835 sd-journal: make journal_file_read_tail_timestamp() notify to the caller that some new journal entries added
Tiny optimization for journal_file_find_newest_for_boot_id().
2024-02-12 01:27:50 +09:00
Yu Watanabe 99e6f682d3 sd-journal: cache last entry offset and journal file state
When the offset of the last entry object (or last object for journal
files generated by an old journald) is not changed, the timestamps
should be updated by journal_file_read_tail_timestamp() are unchanged.
So, we can drop to call fstat() in the function.

As, the journal header is always mapped, so we can read the offset and
journal file state without calling fstat.
Still, when the last entry offset is changed, we may need to call fstat()
to read the entry object. But, hopefully the number of fstat() call
can be reduced.
2024-02-12 01:23:16 +09:00
Yu Watanabe 4ebf678f1f Merge pull request #30260 from yuwata/sd-journal-cleanups
sd-journal: further cleanups
2024-02-12 01:07:11 +09:00
Frantisek Sumsal 8ceac11069 Revert "packit: temporarily build systemd without BPF stuff"
The latest Rawhide compose (20240210.n.1) finished successfully and made
it to the mirrors, so let's drop the bpftool workaround.

This reverts commit 8a0ec8852c.
2024-02-11 16:45:03 +01:00
Daan De Meyer 1a0541d44c repart: Add --generate-fstab= and --generate-crypttab= options
These can be used along with two new settings MountPoint= and
EncryptedVolume= to write fstab and crypttab entries to the given
paths respectively in the root directory that repart is operating on.

This is useful to cover scenarios that aren't covered by the
Discoverable Partitions Spec. For example when one wants to mount
/home as a separate btrfs subvolume. Because multiple btrfs subvolumes
can be mounted from the same partition, we allow specifying MountPoint=
multiple times to add multiple entries for the same partition.
2024-02-10 13:07:02 +01:00
Luca Boccassi 1a383a881b Merge pull request #31273 from mrc0mmand/test-55-arch-gcc
test: make the MemoryHigh= limit a bit more generous with sanitizers
2024-02-10 11:37:53 +00:00
Luca Boccassi 769ca9ab55 Merge pull request #31261 from bluca/repart_engine
repart: support engines and providers
2024-02-09 21:28:59 +00:00
Frantisek Sumsal 76aa0d5db1 test: clean up the code a bit 2024-02-09 20:45:47 +01:00
Frantisek Sumsal 974fe6131f test: make the MemoryHigh= limit a bit more generous with sanitizers
When we're running with sanitizers, sd-executor might pull in a
significant chunk of shared libraries on startup, that can cause a lot
of memory pressure and put us in the front when sd-oomd decides to go on
a killing spree. This is exacerbated further on Arch Linux when built
with gcc, as Arch ships unstripped gcc-libs so sd-executor pulls in over
30M of additional shared libs on startup:

~# lddtree build-san/systemd-executor
build-san/systemd-executor (interpreter => /lib64/ld-linux-x86-64.so.2)
    libasan.so.8 => /usr/lib/libasan.so.8
        libstdc++.so.6 => /usr/lib/libstdc++.so.6
        libm.so.6 => /usr/lib/libm.so.6
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1
    libsystemd-core-255.so => /root/systemd/build-san/src/core/libsystemd-core-255.so
        libaudit.so.1 => /usr/lib/libaudit.so.1
            libcap-ng.so.0 => /usr/lib/libcap-ng.so.0
...
    libseccomp.so.2 => /usr/lib/libseccomp.so.2
    libubsan.so.1 => /usr/lib/libubsan.so.1
    libc.so.6 => /usr/lib/libc.so.6

~# ls -Llh /usr/lib/libasan.so.8 /usr/lib/libstdc++.so.6 /usr/lib/libubsan.so.1
-rwxr-xr-x 1 root root 9.7M Feb  2 10:36 /usr/lib/libasan.so.8
-rwxr-xr-x 1 root root  21M Feb  2 10:36 /usr/lib/libstdc++.so.6
-rwxr-xr-x 1 root root 3.2M Feb  2 10:36 /usr/lib/libubsan.so.1

Sanitized libsystemd-core.so is also quite big:

~# ls -Llh /root/systemd/build-san/src/core/libsystemd-core-255.so /usr/lib/systemd/libsystemd-core-255.so
-rwxr-xr-x 1 root root  26M Feb  8 19:04 /root/systemd/build-san/src/core/libsystemd-core-255.so
-rwxr-xr-x 1 root root 5.9M Feb  7 12:03 /usr/lib/systemd/libsystemd-core-255.so
2024-02-09 20:45:39 +01:00
Daan De Meyer 23f9af646b Merge pull request #29960 from CodethinkLabs/vmspawn/mkosi-features
vmspawn: add features required by mkosi
2024-02-09 18:39:54 +01:00
Lennart Poettering e90ec3a847 Merge pull request #31269 from poettering/vconsole-enodev
vconsole/pid1: handle ENODEV on /dev/console somewhat graceful
2024-02-09 18:04:09 +01:00
Luca Boccassi 7716498548 Merge pull request #31268 from poettering/bpf-device-fixes
bpf-devices: various smaller fixes
2024-02-09 16:34:52 +00:00
Yu Watanabe 8c244a5094 systemctl: fix typo
Follow-up for 1baa0415ae.
2024-02-10 00:34:08 +09:00
Yu Watanabe 07ae001f81 test: add tests for journal_file_next_entry() 2024-02-10 00:30:04 +09:00
Yu Watanabe 6c4ea35962 sd-journal: do not read unnecessary object
In journal_file_next_entry(), if the passed offset matches an entry object,
then generic_array_bisect() returns the object, but the object we
requested is the next (or previous) object. Hence, we should not validate
the object returned by generic_array_bisect(), otherwise it may fail
when the journal is corrupted.

Note the validity of the entry object that should be returned by
journal_file_next_entry() will be checked in the following generic_array_get().
So, when journal_file_next_entry() succeeds, the returned object is
always validated.
2024-02-10 00:29:16 +09:00
dependabot[bot] 0279c0abf3 build(deps): bump systemd/mkosi
Bumps [systemd/mkosi](https://github.com/systemd/mkosi) from dbce89aabda438ba58080366631b2c242e365f21 to 070528fec478fc93af7ec057a5d2fd0045123c99.
- [Release notes](https://github.com/systemd/mkosi/releases)
- [Changelog](https://github.com/systemd/mkosi/blob/main/NEWS.md)
- [Commits](https://github.com/systemd/mkosi/compare/dbce89aabda438ba58080366631b2c242e365f21...070528fec478fc93af7ec057a5d2fd0045123c99)

---
updated-dependencies:
- dependency-name: systemd/mkosi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-02-09 16:28:12 +01:00
Yu Watanabe 1b4e322cdb sd-journal: always put verified object into the chain cache
Let's consider the case that
- the first array contains valid entries,
- all entries in the second array are corrupted.

Then, when we are going to upwards, and a call of generic_array_bisect()
matches the last entry of the first array, then the second array was
cached with last_index == UINT64_MAX, instead of the first array with
its last entry.
Hence, when generic_array_bisect() is called next time, the function call
of test() always fail. So, the cache entry is mostly meaningless.

Let's always store valid cache entry.
2024-02-10 00:26:17 +09:00
Yu Watanabe 821d55a8d7 sd-journal: drop duplicated tests in generic_array_bisect()
When left == right (== 0), generic_array_bisec_step() does not update
left and right. So the following test is exactly the same.
2024-02-10 00:26:17 +09:00
Luca Boccassi 0a8264080a repart: support OpenSSL engines/providers for signing
The provider API which is new requires providers, which are not
widely available and don't work very well yet, so also use a
fallback with the legacy engine API.
2024-02-09 15:11:25 +00:00
Luca Boccassi dba0afa14e openssl: add helper to load key from provider/engine
It's not the literal private key, but EVP_PKEY becomes a reference
to the engine/provider that OpenSSL knows how to use later
2024-02-09 14:36:10 +00:00
Lennart Poettering 3a51cf673f bpf-devices: if a device node is referenced which doesn't exist, downgrade log message
Currently in many of our test cases you'll see a warning about a tun
device not being around. Let's make that quiet, since if there's no such
device there's no point in adding it to a policy anyway, and it makes
useless noise go away.

We keep the warning as a warning if a device node is missing for other
errors than ENOENT.
2024-02-09 15:32:10 +01:00
Lennart Poettering 958b73bef1 bpf-devices: normalize the return handling of functions that put together policy
under some conditions we suppress generating BPF programs. Let's
systematically return 0 when we do this, and 1 if we did actually
soething, instead of second guessing this in the caller.

This is not only more correct, but allows us to suppress BPF programs in
more cases in later commits.
2024-02-09 15:32:10 +01:00
Lennart Poettering 893220f62f bpf-devices: normalize how we pass around major/minor values
There's some unclarity whether major/minor of device nodes are supposed
to be "unsigned" or "dev_t". Various codebases assume the latter, but
glibc's major()/minor() types actually return a value typed to
"unsigned". On glibc dev_t is actually 64bit even if the kernel only
exposes 32bit. Hence this distinction kinda matters.

Let's clean things up a bit with handling: let's followe glibc's type
system here, and use unsigned (and not int).

Also let's pass invalid major/minor values around as UINT_MAX rather
than via pointers, to match how we usually do this, and to shorten our
code a bit. This is safe, since given the linux dev_t space being 32bit
only we can't possibly have a valid major or minor this hight, given
they must be smaller in size. While other archs disagree on the types of
major/minor, they also tend to have similar limits. In fact on FreeBSD
for example major()/minor() returns a signed int. Which would hence also
mean that UINT_MAX cannot be a valid major or minor.
2024-02-09 15:32:04 +01:00
Luca Boccassi c505275476 Merge pull request #31243 from YHNdnzj/systemctl-disable-now-template
systemctl: support disable/mask --now with unit template
2024-02-09 14:29:50 +00:00
Luca Boccassi 9131b93590 Merge pull request #31224 from mrc0mmand/packit-bpftool-workaround
packit: temporarily build systemd without BPF stuff
2024-02-09 13:48:13 +00:00
Luca Boccassi 6e7134ad95 Merge pull request #31264 from poettering/sysext-help
sysext: tweaks to the systemd-sysext/systemd-confext --help text
2024-02-09 13:46:28 +00:00
Lennart Poettering f244e7a7ea pid1,vconsole-setup: gracefully handle if /dev/vconsole is not accessible due to ENODEV
I think this is generally the right thing to do and is just an extension
of the existing ENOENT check.

Prompted by: #31257
2024-02-09 14:19:20 +01:00
Lennart Poettering a0f6b68135 dev-setup: normalize logging around lock_dev_console()
Previously this function would log loudly in some cases but not in
others. Clean this up, and dont log at all, matching our coding style
which says we should either log in all error cases or in none.

Both callers of this function do logging already, hence no need to
duplicate it here.
2024-02-09 14:19:16 +01:00
Sam Leonard 51747b3444 vmspawn: accept kvm/vhost-vsock device fds through sd_listen 2024-02-09 12:46:44 +00:00
Sam Leonard 75331bedeb vmspawn: add nic configuration 2024-02-09 12:46:43 +00:00
Sam Leonard 88af28d170 vmspawn: add initrd configuration option 2024-02-09 12:46:43 +00:00
Daan De Meyer 317cb6f9b5 Install pacman in Arch Linux image
We install apt and dnf in the other images as well, so lets be
consistent and install pacman in the Arch image as well.
2024-02-09 12:41:54 +00:00
Frantisek Sumsal e7a52b1b33 test: adjust test-path to fail gracefully with the new pidfd_spawn stuff
Since 2e106312e2 the test unit fails with 'resources' result instead of
'exit-code', which the test didn't account for when running unprivileged.

Before 2e106312e2:
$ /root/systemd/build/test-path
Failed to start transient scope unit: Interactive authentication required.
Couldn't allocate a scope unit for this test, proceeding without.
...
-.slice: Failed to enable/disable controllers on cgroup /user.slice/user-1000.slice/session-1.scope, ignoring: Permission denied
app.slice: Failed to create cgroup /user.slice/user-1000.slice/session-1.scope/app.slice: Permission denied
-.slice: Failed to enable/disable controllers on cgroup /user.slice/user-1000.slice/session-1.scope, ignoring: Permission denied
app.slice: Failed to create cgroup /user.slice/user-1000.slice/session-1.scope/app.slice: Permission denied
...
line 151: path-exists.path: state = running; result = success (left: 29986250)
line 151: path-exists.service: state = start; result = success
path-exists.service: Main process exited, code=exited, status=219/CGROUP
path-exists.service: Failed with result 'exit-code'.
line 151: path-exists.path: state = running; result = success (left: 29985948)
line 151: path-exists.service: state = failed; result = exit-code
Failed to start service path-exists.service, aborting test: failed/exit-code

After 2e106312e2:
$ /root/systemd/build/test-path
Failed to start transient scope unit: Interactive authentication required.
Couldn't allocate a scope unit for this test, proceeding without.
...
-.slice: Failed to enable/disable controllers on cgroup /user.slice/user-1000.slice/session-1.scope, ignoring: Permission denied
app.slice: Failed to create cgroup /user.slice/user-1000.slice/session-1.scope/app.slice: Permission denied
-.slice: Failed to enable/disable controllers on cgroup /user.slice/user-1000.slice/session-1.scope, ignoring: Permission denied
app.slice: Failed to create cgroup /user.slice/user-1000.slice/session-1.scope/app.slice: Permission denied
path-exists.service: Failed to spawn executor: No such file or directory
path-exists.service: Failed to spawn 'start' task: No such file or directory
path-exists.service: Failed with result 'resources'.
2024-02-09 12:50:21 +01:00
Frantisek Sumsal 8a0ec8852c packit: temporarily build systemd without BPF stuff
The kernel-tools meta-package was retired in Rawhide, but its
replacement has not landed, yet. Until that happens, let's build without
the bpf-framework stuff.
2024-02-09 12:50:21 +01:00
Sam Leonard 19301e76e6 vmspawn: synthesise root= argument for direct kernel boot 2024-02-09 11:43:19 +00:00
Sam Leonard 0f25e3e443 vmspawn: add kernel configuration options 2024-02-09 11:43:19 +00:00
Sam Leonard cf3beb273b vmspawn: add swtpm feature 2024-02-09 11:43:18 +00:00
Sam Leonard fd1cd4a843 path-lookup: add runtime_directory for resolving $RUNTIME_DIRECTORY 2024-02-09 11:43:18 +00:00
Daan De Meyer 7bf52f5d1c Add systemd.default_debug_tty=
Let's allow configuring the debug tty independently of enabling/disabling
the debug shell. This allows mkosi to configure the correct tty while
leaving enabling/disabling the debug tty to the user.
2024-02-09 11:47:53 +01:00
Frantisek Sumsal c078f4af6b semaphore: temporarily pin autopkgtest to v5.32
The latest commit (ATTOW) [0] calls adduser with --logmsglevel, which is
not a valid flag for adduser on Ubuntu Focal/Jammy.

[0] https://salsa.debian.org/ci-team/autopkgtest/-/commit/9c033b3db453acaa103bae03a4a5dcebe3858089
2024-02-09 11:18:25 +01:00
Yu Watanabe 6a32108812 man: add missing "=" after setting name
Follow-up for b93bf1bf9f.
2024-02-09 19:03:56 +09:00
Frantisek Sumsal 8632590278 Merge pull request #31230 from mrc0mmand/test-seccomp
process-util: use only the least significant byte from personality()
2024-02-09 10:39:23 +01:00
Yu Watanabe 15875f524f README: update link to CentOS CI 2024-02-09 18:08:58 +09:00
Yu Watanabe d2eb27eb7b man: fix typo
Follow-up for 631cf7f004.
2024-02-09 17:49:48 +09:00
Lennart Poettering 00d0f58443 sysext: output an appropriate blurb for the mode we are executed in 2024-02-09 09:41:33 +01:00
Lennart Poettering 82d371eeaf sysext: point to the right man page for the mode we are invoked in 2024-02-09 09:41:26 +01:00
Lennart Poettering 457a74b3d0 sysext: rename "directory_name" field to "full_identifier"
So the field contains simply the full name of the command being invoked,
hence rename the field to match the contents, and to mirror the
"short_identifier" field.

Interestingly, the field is apparently not actually used by anything
though! But we are not going to remove it, since a follow-up commit will
start making use of it.
2024-02-09 09:41:19 +01:00