Combining --ephemeral with --grow-image= truncated the file passed to
--image= to the requested size before the VM was started, so a mode whose
entire purpose is to leave the original image untouched modified it on
disk.
Skip growing the original image when running ephemeral, and instead create
the qcow2 overlay with the requested virtual size, so the guest sees the
larger disk while the base image stays read-only. As only the overlay is
sized in that case, the base image format no longer matters, so allow
--grow-image= together with qcow2 images when running ephemeral. The round
up to a multiple of 4096 moves to option parsing so both paths apply the
same value.
The vmspawn drive test grows an ephemeral VM and checks that the image
passed to --image= keeps its size, and that the overlay QEMU runs off is
created at the requested one.
CoCo firmware is stateless, so Secure Boot keys cannot be enrolled at
runtime: it only enforces Secure Boot with keys baked in at build time,
refusing unsigned images from the first boot. The default exclusion of
the enrolled-keys firmware feature hence keeps unsigned images bootable,
but it also makes firmware discovery fail on distros that only ship
SNP/TDX firmware with pre-enrolled keys (e.g. Fedora's TDVF). Drop the
rejection of --secure-boot=yes with --coco= and instead treat it as an
opt-in to such firmware, by lifting the enrolled-keys exclusion.
While at it, reject --efi-nvram-template= and an explicit
--efi-nvram-state= path with --coco=, which were silently ignored, as
stateless firmware has no NVRAM to instantiate or persist.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Instead of requiring --firmware= to point at a raw .fd image for
--coco=sev-snp/tdx, pick a suitable firmware automatically from the
QEMU firmware descriptors, requiring the amd-sev-snp/intel-tdx feature.
CoCo firmware is stateless (it carries no NVRAM template), so teach
find_ovmf_config() to select stateless raw-format firmware via new
flags. --firmware= now uniformly takes a firmware descriptor path.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
SMBIOS OEM strings are host-controlled and normally distrusted by PID1
in confidential VMs. Under TDX, however, the SMBIOS table (including
Type 11) is measured into RTMR0 by the firmware (TDVF), so a remote
verifier can detect host tampering with credentials delivered this
way. Accept them there, while keeping fw_cfg distrusted as those
items are not measured even on TDX.
This lets systemd-vmspawn deliver credentials to TDX guests via the
normal SMBIOS path, unlike SNP which requires the initrd cpio channel.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
To query a TD quote, the TDX guest must connect to the Quote Generation
Service (QGS), a SGX enclave running on the host. We check if the
service is exposed via a well-known unix socket, then pass that socket
or a fallback well-known vsock address to QEMU.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Wire up --coco=tdx alongside the existing SEV-SNP path. TDX requires KVM
on x86_64, a raw TDVF firmware loaded via -bios (no pflash/NVRAM split),
kernel-irqchip=split, and the "host" CPU model since QEMU rejects named
models. Sets up the tdx-guest object and confidential-guest-support=tdx0.
TDX measurement is different from QEMU's kernel-hashes injection: TDX
provides runtime measurements via RTMRs, so the initial measurement only
covers the firmware, which then measures the rest of the boot chain into
those RTMRs (done by OVMF today). Therefore a restriction to direct
kernel boot isn't required either.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Under --coco=sev-snp, credentials no longer flow through SMBIOS/fw_cfg
(which the guest PID1 discards as unmeasured in confidential VMs) but
through a cpio archive appended to the initrd, landing in the @system
bucket via the new /.extra/system_credentials/ initrd path. Update
systemd-vmspawn(1) to describe this and the guest systemd version
requirement.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
This has little to do with host configuration (where it was so far), and
a lot with what being executed, let's move it over.
Note that --help and man page so far differed here quite a bit: the
former had the "Execution" section, the latter didn't. This creates it
in the man page, to bring the two back in sync.
Add --confidential-computing=sev-snp to run the guest as an AMD SEV-SNP
confidential VM. Loads a raw OVMF firmware blob via -bios (SNP doesn't
support the pflash + NVRAM split), attaches a sev-snp-guest object,
and hashes the kernel, initrd and cmdline into the launch measurement
when direct kernel boot is used. Incompatible features (Secure Boot,
CXL, virtio-balloon, SMBIOS credentials) are rejected or disabled; an
attached vTPM must be treated as untrusted by the guest.
The feature is marked experimental in the man page.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
- Document the new --bind-volume= option in systemd-vmspawn(1) and
the new bind-volume / unbind-volume verbs in machinectl(1).
- Add an integration test
(TEST-87-AUX-UTILS-VM.bind-volume.sh) covering boot-time attach
via --bind-volume, runtime attach via 'machinectl bind-volume',
runtime detach via 'machinectl unbind-volume', the StorageImmutable
rejection of attempts to detach boot-time volumes, and the
NoSuchStorage rejection of detach on unknown names.
- Strike "hook-up in systemd-vmspawn" from TODO.md; the nspawn and
service-manager hookups remain.
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Add options to vmspawn to configure journal-remote disk usage limits
when forwarding journal entries from the VM. These are passed through
as --max-use=, --keep-free=, --max-file-size=, and --max-files=
command-line arguments to systemd-journal-remote.
Add --max-use=, --keep-free=, --max-file-size=, and --max-files=
command-line options to systemd-journal-remote to allow overriding the
corresponding settings from the configuration file.
Add $SYSTEMD_JOURNAL_REMOTE_CONFIG_FILE environment variable support
to systemd-journal-remote. When set, the specified file is used
instead of the default configuration file and drop-in directories.
When set to the empty string or /dev/null, configuration file parsing
is skipped entirely. vmspawn sets this to /dev/null in the child
process to avoid inheriting the host's journal-remote configuration.
Make fork_notify() argv parameter optional. When NULL is passed,
fork_notify() returns 0 in the child (with $NOTIFY_SOCKET set) and
lets the caller run custom code before exec. Returns 1 in the parent.
This allows vmspawn to set environment variables in the child without
polluting the parent process.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
When --linux= specifies a non-PE kernel image, automatically disable
UEFI firmware loading (as if --firmware= was passed). If --firmware=
is explicitly set to a path in this case, fail with an error. Booting
a UKI with --firmware= is also rejected since UKIs require UEFI.
--firmware= (empty string) can also be used explicitly to disable
firmware loading for PE kernels.
Other changes:
- Extract OVMF pflash drive setup into cmdline_add_ovmf()
- Extract kernel image type detection into determine_kernel()
- Add smbios_supported() helper to centralize the SMBIOS availability
check (always available on x86, elsewhere requires firmware)
- Gate SMM, OVMF drives, SMBIOS11 and credential SMBIOS paths
on firmware/SMBIOS being available
- Beef up the credential logic to fall back to fw_cfg and kernel
command line in case SMBIOS is not available
Drop the --cxl= option and unconditionally enable cxl=on the QEMU
machine type whenever the host architecture supports it (x86_64 and
aarch64). The flag was only added for testing parity with mkosi's CXL=
setting and there is no reason to leave it as an opt-in toggle: with no
pxb-cxl device or cxl-fmw window attached, enabling it on the machine
only reserves a small MMIO region and emits an empty CEDT, so the cost
is negligible while removing one knob users would otherwise have to
flip explicitly to exercise the CXL code paths in QEMU.
Move register_machine() and unregister_machine() from
vmspawn-register.{c,h} into shared machine-register.{c,h} so both
nspawn and vmspawn can use the same implementation.
The unified register_machine() uses varlink first (for richer
features like SSH support and unit allocation) with a D-Bus
RegisterMachineWithNetwork fallback for older machined. The
interface adds a class parameter ("vm" or "container") and
local_ifindex for nspawn's network interface support.
The unified unregister_machine() similarly tries varlink first
(io.systemd.Machine.Unregister) before falling back to D-Bus.
Both register_machine() and unregister_machine() only log at debug
level internally, leaving error/notice logging to callers.
Add register_machine_with_fallback() which tries system and/or user
scope registration based on a RuntimeScope parameter
(_RUNTIME_SCOPE_INVALID for both), and
unregister_machine_with_fallback() as its counterpart. Both use
RET_GATHER() to collect errors from each scope.
Make --register= a tristate (yes/no/auto) defaulting to auto. When
set to auto, registration failures are logged at notice level and
ignored. When set to yes, failures are fatal.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Switch the negation character for firmware feature exclusion from
"!" to "~" to be consistent with other systemd options that support
negation such as SystemCallFilter=.
Add --cxl=BOOL option to enable CXL (Compute Express Link) support in
the virtual machine. CXL is a high-speed interconnect standard that
allows CPUs to access memory attached to devices such as accelerators
and memory expanders, enabling flexible memory pooling and expansion
beyond what is physically installed on the motherboard. When enabled,
adds cxl=on to the QEMU machine configuration. Only supported on x86_64
and aarch64 architectures.
This is added for testing purposes and for feature parity with mkosi's
CXL= setting.
Extend --ram= to accept an optional maximum size for memory hotplug,
using the syntax --ram=SIZE[:MAXSIZE] (e.g. --ram=2G:8G). When a
maximum is specified, the maxmem key is added to the QEMU memory
configuration section to enable memory hotplug up to the given limit.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
Add a --console-transport= option that selects between virtio-serial
(the default, appearing as /dev/hvc0) and a regular serial port
(appearing as /dev/ttyS0 or /dev/ttyAMA0 depending on architecture).
This is primarily useful for testing purposes, for example to test
sd-stub's automatic console= kernel command line parameter handling. It
allows verifying that the guest OS correctly handles serial console
configurations without virtio.
When serial transport is selected, -serial chardev:console is used on
the QEMU command line to connect the chardev to the platform's default
serial device. This cannot be done via the QEMU config file as on some
platforms (e.g. ARM) the serial device is a sysbus device that can only
be connected via serial_hd() which is populated by -serial.
Add DISK_TYPE_SCSI_CD to support attaching disk images as CD-ROM
drives, needed for testing El Torito ISO images built by
systemd-repart.
When --image-disk-type=scsi-cd is specified, the image is attached
with media=cdrom and readonly=on on the drive, using scsi-cd as the
device driver on the SCSI bus. This also works for --extra-drive=
with the scsi-cd: prefix.
The QEMU configuration matches the standard OVMF CD-ROM boot setup:
-drive if=none,media=cdrom,format=raw,readonly=on
-device virtio-scsi-pci
-device scsi-cd
When direct kernel booting with scsi-cd, if the kernel command line
contains "rw", append "ro" to override it since CD-ROMs are
read-only.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
It's useful to be able to check what firmware description vmspawn
will select. In particular, this will allow me to figure out the
nvram template file that will be picked up so I can pick it up in
mkosi and operate on it to pass a modified version of it to vmspawn
with --efi-nvram-template=.
Add --efi-nvram-template=PATH to specify a custom firmware variables
file to copy and use as the initial EFI NVRAM state instead of the
default template from the firmware definition.
Add --firmware-features=FEATURE[,FEATURE...] to require or exclude
specific firmware features during automatic firmware discovery.
Features prefixed with "!" are excluded. If a feature appears in both
the included and excluded lists, inclusion takes priority. Firmware
with the "enrolled-keys" feature is excluded by default.
Refactor --secure-boot= to operate on the firmware features sets
instead of maintaining a separate tristate. --secure-boot=yes adds
"secure-boot" to the include set, --secure-boot=no adds it to the
exclude set, and --secure-boot=auto removes it from both.
Generalize find_ovmf_config() to accept include/exclude feature sets
instead of a secure boot tristate, removing the special-cased
enrolled-keys and secure-boot filtering logic.
Co-developed-by: Claude Opus 4.6 <noreply@anthropic.com>
vmspawn previously hardcoded virtio-blk for all drives. This adds
--image-disk-type= to select the root disk type (virtio-blk,
virtio-scsi, or nvme) and allows per-drive overrides via a
colon-separated prefix on --extra-drive=. The format and disk type
prefixes can appear in any order since their value sets don't overlap.
For virtio-scsi, a single shared controller is created with drives
attached as scsi-hd devices. For nvme, each drive gets its own
controller. Both have serial number length limits (30 and 20 characters
respectively), so long filenames are replaced with a truncated SHA-256
hex digest.
Extend --image-disk-type= and the --extra-drive= disk type prefix to
support nvme in addition to virtio-blk and virtio-scsi:
systemd-vmspawn --image-disk-type=nvme --image=image.raw
systemd-vmspawn --image=image.raw --extra-drive=nvme:data.raw
The NVMe serial number is limited to 20 characters by the NVMe spec.
If the image filename exceeds this, it is hashed with SHA-256 and
truncated to 20 hex characters via the disk_serial() helper introduced
in the previous commit.
Signed-off-by: Christian Brauner <brauner@kernel.org>
Add --image-disk-type= to select the disk type for the root disk, and
allow specifying the disk type as a colon-separated prefix on
--extra-drive=:
systemd-vmspawn --image-disk-type=virtio-scsi --image=image.raw
systemd-vmspawn --image=image.raw --extra-drive=virtio-scsi:data.raw
For --extra-drive=, the format and disk type prefixes can appear in any
order since the value sets don't overlap:
--extra-drive=raw:virtio-scsi:/path
--extra-drive=virtio-scsi:raw:/path
Extra drives inherit --image-disk-type= by default unless overridden
with an explicit prefix.
vmspawn originally used virtio-scsi for all drives but switched to
virtio-blk in 1f24a954e4 for simplicity and direct kernel boot
compatibility. This makes virtio-scsi available again as an explicit
option for cases where a SCSI storage topology is desired.
For virtio-scsi, a shared virtio-scsi-pci controller is created and
drives are attached as scsi-hd devices. The SCSI serial number is
limited to 30 characters, so filenames exceeding this are hashed with
SHA-256.
Signed-off-by: Christian Brauner <brauner@kernel.org>
Closes#40877. As requested, --extra-drive=path[:format] is changed
to --extra-drive=[format:]path, so that the parsing is less ambiguous.
(In the original request, it was requested that the empty format can be
used also, but that was dropped in the second version of the patch.)
This patch adds a `--ephemeral` option to vmspawn that will turn
on snapshot mode on the qemu disk passed via `--image` or the
directory passed via `--directory`. For disk images it uses the
native mechanism that qemu provides. For directories it (re)uses
the snapshot mechanism that nspawn is using, i.e. it will create
a btrfs snapshot if possible and if not falls back to a traditional
directory copy.
The systemd-vmspawn man page has a note that the `--notify-ready`
defaults to "true" and then says that this is unlike "systemd-vmspawn"
where it defaults to "false". This appears to be a copy/paste issue
and here we want "unlike systemd-nspawn".
The UX of registering with the user session machined
instance is much better as there won't be an authorization
prompt. To make that available for users, let's add --user
and --system switches for vmspawn. For backwards compat, we'll
still try to register with the system machined instance if the
user machined instance is not available.
Found using linkchecker.
For virtiofsd, the man page is maintained upstream, but doesn't seem to be
available in any of the usual places. So let's link to the Debian version.
systemd.filter I have no idea what it is.
This mimics the switch of the same name from nspawn: it controls whether
we expect a READY=1 message from the payload or not. Previously we'd
always expect that. This makes it configurable, just like it is in
nspawn.
There's one fundamental difference in behaviour though: in nspawn it
defaults to off, in vmspawn it defaults to on. (for historical reasons,
ideally we'd default to on in both cases, but changing is quite a compat
break both directly and indirectly: since timeouts might get triggered).
This beefs up the cgroup logic, adding --slice=, --property= to vmspawn
the same way it already exists in nspawn.
There are a bunch of differences though: we don't delegate the cgroup
access in the allocated unit (since qemu wouldn't need that), and we do
registration via varlink not dbus. Hence, while this follows a similar
logic now, it differs in a lot of details.
This makes in particular one change: when invoked on the command line
we'll only add the qemu instance to the allocated scope, not the vmspawn
process itself (this follows more closely how nspawn does this where
only the container payload has its scope, not nspawn itself). This is
quite tricky to implement: unlike in nspawn we have auxiliary services
to start, with depencies to the scope. This means we need to start the
scope early, so that we know the scope's name. But the command line to
invoke is only assembled from the data we learn about the auxiliary
services, hence much later. To addres we'll now fork off the child that
eventually will become early, then move it to a scope, prepare the
cmdline and then very late send the cmdline (and the fds we want to
pass) to the prepared child, which then execs it.
When using vmspawn on particleos image we really want that the TPM state
is retained between invocation, since the encryption key is locked to
the TPM after all. Hence let's support that.
This adds --tpm-state= which can be used to configure a path to store
the TPM state in. It can also be used to force tpm state to be transient
or to let vmpsawn pick the path automatically.
While we are at it, let's also revamp the runtime dir handling in
vmspawn: let's no longer place the sockets the auxiliary services listen
on within their own runtime directories. Instead, just drop the runtime
directories for them entirely (since neither virtiofsd, nor swtpm
actually use them). Also, let systemd clean up the sockets
automatically.
Daan De Meyer
Paul Meyer
Lennart Poettering
albertescanes
Christian Brauner
Zbigniew Jędrzejewski-Szmek
Mike Yuan
Michael Vogt
Kai Lüke
Michael Vogt
Daan De Meyer
Lennart Poettering
Zbigniew Jędrzejewski-Szmek
Yu Watanabe
Jan Engelhardt