The usual approach is to put 'addopts = --flakes' in setup.cfg. Unfortunately
this fails badly when pytest-flakes is not installed:
ERROR: usage: test_ukify.py [options] [file_or_dir] [file_or_dir] [...]
test_ukify.py: error: unrecognized arguments: --flakes
pytest-flakes is not packaged everywhere, and this test is not very important,
so let's just do it only if pytest-flakes is available. We now detect if
pytest-flakes is available and only add '--flakes' conditionally. This
unfortunately means that when invoked via 'pytest' or directly as
'src/ukify/test/test_ukify.py', '--flakes' will not be appended automatically.
But I don't see a nice way to achieve previous automatic behaviour.
(I first considered making 'setup.cfg' templated. But then it is created
in the build directory, but we would need it in the source directory for
pytest to load it automatically. So to load the file, we'd need to give an
argument to pytest anyway, so we don't gain anything with this more complex
approach.)
Note to self: PEP 585 introduced using collection types as types,
and is available since 3.9. PEP 604 allows writing unions with "|",
but is only available since 3.10, so not yet here because we maintain
compat with 3.9.
In some ways this is similar to mkosi: we have a argparse.ArgumentParser()
with a bunch of options, and a configparser.ConfigParser() with an
overlapping set of options. Many options are settable in both places, but
not all. In mkosi, we define this in three places (a dataclass, and a
function for argparse, and a function for configparser). Here, we have one
huge list of ConfigItem instances. Each instance specifies the full metadata
for both parsers. Argparse generates a --help string for all the options,
and we also append a config file sample to --help based on the ConfigItem
data:
$ python src/ukify/ukify.py --help|tail -n 25
config file:
[UKI]
Linux = LINUX
Initrd = INITRD…
Cmdline = TEXT|@PATH
OSRelease = TEXT|@PATH
DeviceTree = PATH
Splash = BMP
PCRPKey = KEY
Uname = VERSION
EFIArch = ia32|x64|arm|aa64|riscv64
Stub = STUB
PCRBanks = BANK…
SigningEngine = ENGINE
SecureBootPrivateKey = SB_KEY
SecureBootCertificate = SB_CERT
SignKernel = SIGN_KERNEL
[PCRSignature:NAME]
PCRPrivateKey = PATH
PCRPublicKey = PATH
Phases = PHASE-PATH…
While writing this I needed to check the argument parsing, so I added
a --summary switch. It just pretty-prints the resulting option dictionary:
$ python src/ukify/ukify.py /efi//3a9d668b4db749398a4a5e78a03bffa5/6.2.11-300.fc38.x86_64/linux /efi//3a9d668b4db749398a4a5e78a03bffa5/6.2.11-300.fc38.x86_64/initrd --pcr-private-key=PRIV.key --pcr-public-key=PUB.key --config=man/ukify-example.conf --summary
Host arch 'x86_64', EFI arch 'x64'
{'_groups': [0, 'initrd', 'system'],
'cmdline': 'A1 B2 C3',
'config': 'man/ukify-example.conf',
'devicetree': None,
'efi_arch': 'x64',
'initrd': [PosixPath('initrd1'),
PosixPath('initrd2'),
PosixPath('initrd3'),
PosixPath('/efi/3a9d668b4db749398a4a5e78a03bffa5/6.2.11-300.fc38.x86_64/initrd')],
'linux': PosixPath('/efi/3a9d668b4db749398a4a5e78a03bffa5/6.2.11-300.fc38.x86_64/linux'),
'measure': None,
'os_release': PosixPath('/etc/os-release'),
'output': 'linux.efi',
'pcr_banks': ['sha1', 'sha384'],
'pcr_private_keys': [PosixPath('PRIV.key'),
PosixPath('pcr-private-initrd-key.pem'),
PosixPath('pcr-private-system-key.pem')],
'pcr_public_keys': [PosixPath('PUB.key'),
PosixPath('pcr-public-initrd-key.pem'),
PosixPath('pcr-public-system-key.pem')],
'pcrpkey': None,
'phase_path_groups': [None,
['enter-initrd'],
['enter-initrd:leave-initrd',
'enter-initrd:leave-initrd:sysinit',
'enter-initrd:leave-initrd:sysinit:ready']],
'sb_cert': PosixPath('mkosi.secure-boot.crt'),
'sb_key': PosixPath('mkosi.secure-boot.key'),
'sections': [],
'sign_kernel': None,
'signing_engine': None,
'splash': None,
'stub': PosixPath('/usr/lib/systemd/boot/efi/linuxx64.efi.stub'),
'summary': True,
'tools': None,
'uname': None}
With --summary, existence of input paths is not checked. I think we'll
want to show them, instead of throwing an error, but in red, similarly to
'bootctl list'.
This also fixes tests which were failing with e.g.
E FileNotFoundError: [Errno 2] No such file or directory: '/ARG1'
=========================== short test summary info ============================
FAILED ../src/ukify/test/test_ukify.py::test_parse_args_minimal - FileNotFoun...
FAILED ../src/ukify/test/test_ukify.py::test_parse_args_many - FileNotFoundEr...
FAILED ../src/ukify/test/test_ukify.py::test_parse_sections - FileNotFoundErr...
=================== 3 failed, 10 passed, 3 skipped in 1.51s ====================
Fixup for 22ad038ac6 and
3fc5eed470. It seems that the tests are
not executed properly in CI. Nevertheless, test-ukify appears in logs:
rpm-build:fedora-rawhide-x86_64:
409/1191 systemd / test-ukify OK 0.16s
This is strange.
We generally nowadays use UPPERCASE for parameters in variuos help text.
Let's be consistent here too, and also drop duplicated 'usage:':
$ ukify -h
usage: ukify [options…] LINUX INITRD…
ukify -h | --help
Build and sign Unified Kernel Images
positional arguments:
LINUX vmlinuz file [.linux section]
INITRD… initrd files [.initrd section]
...
Older stubs are either not stripped, causing their total size to be
unaligned because of an unaligned symbol table at the end, or stripped,
causing the raw data pointers and sizes to be unaligned because strip
does not follow the PE spec correctly when stripping. Let's add
workarounds for both issues, so that we can use ukify with older stubs
as well.
Older versions of the stub are not aligned to the PE file alignment
size. If we remove the assertions, the UKI still boots without issues,
so let's drop the assertions and print a message about it instead.
Uncompressed aarch64 and riscv64 kernels have a different startpoint than x86.
Example output from ukify:
aarch64: NotImplementedError: unknown file format (starts with b'MZ@\xfa')
riscv64: NotImplementedError: unknown file format (starts with b'MZo\x10')
Add check for (b'MZ') to catch both in one call.
Fix:
https://github.com/systemd/systemd/issues/26923
objcopy seems to expect that the offset passed to --change-section-vma
is absolute instead of relative to ImageBase. If this is not accounted
for an invalid image is created that cannot be loaded:
0 .osrel 0000016b 0000000200016000 0000000200016000 00000400 2**2
…
6 .text 0000d242 0000000140001000 0000000140001000 00c6e800 2**4
This isn't an issue with gnu-efi based PE images, but natively created
ones will have a non-zero ImageBase.
Instead of consuming N arguments as tools directories, let's always
only consume one argument per specification of --tools. This avoids
issues where the linux image and initrd are interpreted as tools
directories.
llvm-objcopy works on stubs built for foreign architectures whereas
objcopy doesn't so let's prefer using llvm-objcopy instead of objcopy.
llvm-objcopy automatically sets the virtual address and doesn't provide
an option to set it manually so we only add --change-section-vma when
using objcopy
The default section flags differ between llvm-objcopy and objcopy
so we add a default for the section flags so we make sure all sections
are read-only data unless specified otherwise.
It's much nicer for the user if we fail early instead of doing partial
processing if we cannot read some input. We can't do those checks immediately
from argparse.Parser.parse_args(), because we want to fully process the
commandline first. In particular, even with invalid args, if --help is
specified somewhere, we want to handle that. Thus, we need to delay the checks
after argparse.Parser.parse_args() returns.
Ukify didn't have type annotations on functions, but it probably should.
Jörg's suggested correction included them and we might just as well start here.
The autodetection code is supposed to throw ValueError when it
cannot figure out the version so that we fall back to the next method.
With the patch:
Kernel version not specified, starting autodetection 😖.
Real-Mode Kernel Header magic not found
+ readelf --notes vmlinuz/arm64/vmlinuz-6.0.9-300.fc37.aarch64
readelf: vmlinuz/arm64/vmlinuz-6.0.9-300.fc37.aarch64: Error: Not an ELF file - it has the wrong magic bytes at the start
Found uname version: 6.0.9-300.fc37.aarch64
If given, multiple initrds are concatenated into a temporary file which then
becomes the .initrd section.
It is also possible to give no initrd. After all, some machines boot without an
initrd, and it should be possible to use the stub without requiring an initrd.
(The stub might not like this, but this is something to fix there.)
The approach is based on mkinicpio's autodetection.
This is hacky as hell. Some cases are actually fairly nice: ppc64el images have
a note that contains 'uname -r'. (The note is not uniquely labeled at all, and
only contains the release part instead of the full version-hostname-release
string, and we don't actually care about ppc, and it's very hard to read the
note from Python, but in general that'd be the approach I'd like.)
I opted to simply read and decompress the full linux binary in some cases.
Python doesn't make it easy to do streaming decompression with regexp matching,
and it doesn't seem to matter much: the image decompresses in a fraction of a
second.
Some gymnastics were needed to import ukify as a module. Before the file
was templated, this was trivial: insert the directory in sys.path, call import.
But it's a real pain to import the unsuffixed file after processing. Instead,
the untemplated file is imported, which works well enough for tests and is
very simple.
The tests can be called via pytest:
PATH=build/:$PATH pytest -v src/ukify/test/test_ukify.py
or directly:
PATH=build/:$PATH src/ukify/test/test_ukify.py
or via the meson test machinery output:
meson test -C build test-ukify -v
or without verbose output:
meson test -C build test-ukify
Zekret files are obfuscated using base64.
The option is added because we have a similar one for kernel-install. This
program requires python, and some people might want to skip it because of this.
The tool is installed in /usr/lib/systemd for now, since the interface might
change.
A template file is used, but there is no .in suffix.
The problem is that we'll later want to import the file as a module
for tests, but recent Python versions make it annoyingly hard to import
a module from a file without a .py suffix. imp.load_sources() works, but it
is deprecated and throws warnings.
importlib.machinery.SourceFileLoader().load_module() works, but is also
deprecated. And the documented replacements are a maze of twisted little
callbacks that result in an empty module.
So let's take the easy way out, and skip the suffix which makes it easy
to import the template as a module after adding the directory to sys.path.
Features:
- adds sections .linux, .initrd, .uname, .osrel, .pcrpkey, .pcrsig, .cmdline, .splash
- multiple initrds can be concatenated
- section flags are set properly (READONLY, DATA or CODE)
- uses systemd-measure to precalculate pcr measurements and create a signed json policy
- the inner linux image will be signed automatically with sbsign if unsigned
- uses sbsign to sign the output image
- offsets are calculated so that sections are placed adjacent, with .linux last
- custom sections are possible
- multiple pcr signing keys can be specified and different boot phase paths can be
signed with different keys
- most things can be overriden (path to tools, stub file, signing keys, pcr banks,
boot phase paths, whether to sign things)
- superficial verification of slash bmp is done
- kernel uname "scraping" from the kernel if not specified (in a later patch)
TODO:
- change systemd-measure to not require a functional TPM2. W/o this, we'd need
to support all banks in the build machine, which is hard to guarantee.
- load signing keys from /etc/kernel/
- supress exceptions, so if something external fails, the user will not see a traceback
- conversion to BMP from other formats
$ sudo /usr/lib/systemd/ukify \
--tools=build/ \
--measure \
/lib/modules/6.0.5-300.fc37.x86_64/vmlinuz \
/boot/08a5690a2eed47cf92ac0a5d2e3cf6b0/6.0.5-300.fc37.x86_64/initrd \
--secureboot-private-key=server.key --secureboot-certificate=server.crt \
--pcr-private-key=tpm2-pcr-private.pem --pcr-public-key=tpm2-pcr-public.pem \
--cmdline='rw quiet' \
--section test:TESTTESTTEST \
--section test2:TESTTESTTEST2 \
--pcr-banks=sha1 \
--uname="$(uname -rv)"
Host arch 'x86_64', efi arch 'x64'
+ sbverify --list /lib/modules/6.0.5-300.fc37.x86_64/vmlinuz
+ build/systemd-measure calculate --linux=/lib/modules/6.0.5-300.fc37.x86_64/vmlinuz --osrel=/etc/os-release --cmdline=/tmp/tmpcmdline_5aufjir --pcrpkey=tpm2-pcr-public.pem --initrd=/boot/08a5690a2eed47cf92ac0a5d2e3cf6b0/6.0.5-300.fc37.x86_64/initrd --bank=sha1
11:sha1=03df5e5243bc002b959d52359fe04e266d0b5ebf
11:sha1=54949b82bae32e80343ff0f01eeeeb75f4c07d3f
11:sha1=0fc62be88aa9c5ad7282aa8adb504f451bcec9df
11:sha1=b71155e7fcd467f7c1696f675e37887032e2eafa
+ build/systemd-measure sign --linux=/lib/modules/6.0.5-300.fc37.x86_64/vmlinuz --osrel=/etc/os-release --cmdline=/tmp/tmpcmdline_5aufjir --pcrpkey=tpm2-pcr-public.pem --initrd=/boot/08a5690a2eed47cf92ac0a5d2e3cf6b0/6.0.5-300.fc37.x86_64/initrd --bank=sha1 --private-key=tpm2-pcr-private.pem --public-key=tpm2-pcr-public.pem
+ objcopy /usr/lib/systemd/boot/efi/linuxx64.efi.stub --add-section .osrel=/etc/os-release --change-section-vma .osrel=0x22000 --add-section .cmdline=/tmp/tmpcmdline_5aufjir --change-section-vma .cmdline=0x23000 --add-section .pcrpkey=tpm2-pcr-public.pem --change-section-vma .pcrpkey=0x24000 --add-section .initrd=/boot/08a5690a2eed47cf92ac0a5d2e3cf6b0/6.0.5-300.fc37.x86_64/initrd --change-section-vma .initrd=0x25000 --add-section .uname=/tmp/tmpuname0v3uzh5r --change-section-vma .uname=0x4009000 --add-section .test=/tmp/tmptestuxve59c8 --change-section-vma .test=0x400a000 --add-section .test2=/tmp/tmptest2_i143p9i --change-section-vma .test2=0x400b000 --add-section .pcrsig=/tmp/tmppcrsigdtcqxz_w --change-section-vma .pcrsig=0x400c000 --add-section .linux=/lib/modules/6.0.5-300.fc37.x86_64/vmlinuz --change-section-vma .linux=0x400d000 /tmp/uki4vsbf7y8
+ sbsign --key server.key --cert server.crt /tmp/uki4vsbf7y8 --output vmlinuz.efi
warning: data remaining[79849520 vs 79866644]: gaps between PE/COFF sections?
warning: data remaining[79849520 vs 79866648]: gaps between PE/COFF sections?
Signing Unsigned original image
Wrote signed vmlinuz.efi
Zbigniew Jędrzejewski-Szmek
Luca Boccassi
Jan Janssen
Daan De Meyer
Tobias Powalowski
Yu Watanabe
Dmitry V. Levin