From cce3d43a2d7475c09cc56923bd78f88b2199a18c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Tue, 5 Dec 2023 20:26:24 +0100 Subject: [PATCH 1/6] docs/ELF_PACKAGE_NOTES: fix formatting Backticks inside of a code block were displayed literally. --- docs/ELF_PACKAGE_METADATA.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/ELF_PACKAGE_METADATA.md b/docs/ELF_PACKAGE_METADATA.md index 5415441cfb..99da8da8d6 100644 --- a/docs/ELF_PACKAGE_METADATA.md +++ b/docs/ELF_PACKAGE_METADATA.md @@ -45,12 +45,10 @@ the original executable, debuginfo and sources, to further facilitate debugging. * Section header -``` -SECTION: `.note.package` -note type: `0xcafe1a7e` -Owner: `FDO` (FreeDesktop.org) +Section: `.note.package`
+`note type`: `0xcafe1a7e`
+Owner: `FDO` (FreeDesktop.org)
Value: a single JSON object encoded as a zero-terminated UTF-8 string -``` * JSON payload From 6e0921045959fd9a02e99244c6bfd9c15eb903d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Thu, 19 Dec 2024 10:50:56 +0100 Subject: [PATCH 2/6] =?UTF-8?q?docs:=20rename=20COREDUMP=5FPACKAGE=5FMETAD?= =?UTF-8?q?ATA=20=E2=86=92=20ELF=5FPACKAGE=5FMETADATA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This document is renamed without any changes in a separate commit. Otherwise, git does not show a diff, which makes it very hard to review what is actually changed in the text. --- TODO | 6 +- docs/COREDUMP.md | 4 +- docs/COREDUMP_PACKAGE_METADATA.md | 2 +- docs/ELF_PACKAGE_METADATA.md | 197 +----------------- docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md | 197 ++++++++++++++++++ 5 files changed, 205 insertions(+), 201 deletions(-) create mode 100644 docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md diff --git a/TODO b/TODO index 99e8f1c723..ba129e1b8d 100644 --- a/TODO +++ b/TODO @@ -689,8 +689,8 @@ Features: control of the attacker. * also include packaging metadata (á la - https://systemd.io/ELF_PACKAGE_METADATA/) in our UEFI PE binaries, using the - same JSON format. + https://systemd.io/PACKAGE_METADATA_FOR_EXECUTABLE_FILES/) in our UEFI PE + binaries, using the same JSON format. * make "bootctl install" + "bootctl update" useful for installing shim too. For that introduce new dir /usr/lib/systemd/efi/extra/ which we copy mostly 1:1 @@ -2616,7 +2616,7 @@ Features: * coredump: - save coredump in Windows/Mozilla minidump format - when truncating coredumps, also log the full size that the process had, and make a metadata field so we can report truncated coredumps - - add examples for other distros in ELF_PACKAGE_METADATA + - add examples for other distros in PACKAGE_METADATA_FOR_EXECUTABLE_FILES * support crash reporting operation modes (https://live.gnome.org/GnomeOS/Design/Whiteboards/ProblemReporting) diff --git a/docs/COREDUMP.md b/docs/COREDUMP.md index ce58f16028..b632733b70 100644 --- a/docs/COREDUMP.md +++ b/docs/COREDUMP.md @@ -68,8 +68,8 @@ specified service unit, and thus can take benefit of regular service resource management and sandboxing. The `systemd-coredump` handler will extract a backtrace and -[ELF packaging metadata](/ELF_PACKAGE_METADATA) from any coredumps it -receives and log both. +[ELF packaging metadata](/PACKAGE_METADATA_FOR_EXECUTABLE_FILES) +from any coredumps it receives and log both. The information about coredumps stored in the journal can be enumerated and queried with the [`coredumpctl`](https://www.freedesktop.org/software/systemd/man/coredumpctl.html) tool, for example for directly invoking a debugger such as `gdb` on a collected diff --git a/docs/COREDUMP_PACKAGE_METADATA.md b/docs/COREDUMP_PACKAGE_METADATA.md index d84e269404..90d86e9ab8 100644 --- a/docs/COREDUMP_PACKAGE_METADATA.md +++ b/docs/COREDUMP_PACKAGE_METADATA.md @@ -1,4 +1,4 @@ --- layout: forward -target: /ELF_PACKAGE_METADATA +target: /PACKAGE_METADATA_FOR_EXECUTABLE_FILES --- diff --git a/docs/ELF_PACKAGE_METADATA.md b/docs/ELF_PACKAGE_METADATA.md index 99da8da8d6..90d86e9ab8 100644 --- a/docs/ELF_PACKAGE_METADATA.md +++ b/docs/ELF_PACKAGE_METADATA.md @@ -1,197 +1,4 @@ --- -title: Package Metadata for ELF Files -category: Interfaces -layout: default -SPDX-License-Identifier: LGPL-2.1-or-later +layout: forward +target: /PACKAGE_METADATA_FOR_EXECUTABLE_FILES --- - -# Package Metadata for Core Files - -*Intended audience: hackers working on userspace subsystems that create ELF binaries -or parse ELF core files.* - -## Motivation - -ELF binaries get stamped with a unique, build-time generated hex string identifier called `build-id`, -[which gets embedded as an ELF note called `.note.gnu.build-id`](https://fedoraproject.org/wiki/Releases/FeatureBuildId). -In most cases, this allows a stripped binary to be associated with its debugging information. -It is used, for example, to dynamically fetch DWARF symbols from a debuginfo server, or -to query the local package manager and find out the package metadata or, again, the DWARF -symbols or program sources. - -However, this usage of the `build-id` requires either local metadata, usually set up by -the package manager, or access to a remote server over the network. Both of those might -be unavailable or forbidden. - -Thus it becomes desirable to add additional metadata to a binary at build time, so that -`systemd-coredump` and other services analyzing core files are able to extract said -metadata simply from the core file itself, without external dependencies. - -## Implementation - -This document will attempt to define a common metadata format specification, so that -multiple implementers might use it when building packages, or core file analyzers, and -so on. - -The metadata will be embedded in a single, new, 4-bytes-aligned, allocated, 0-padded, -read-only ELF header section, in a name-value JSON object format. Implementers working on parsing -core files should not assume a specific list of names, but parse anything that is included -in the section, and should look for the note using the `note type`. Implementers working on -build tools should strive to use the same names, for consistency. The most common will be -listed here. When corresponding to the content of os-release, the values should match, again for consistency. - -If available, the metadata should also include the debuginfod server URL that can provide -the original executable, debuginfo and sources, to further facilitate debugging. - -* Section header - -Section: `.note.package`
-`note type`: `0xcafe1a7e`
-Owner: `FDO` (FreeDesktop.org)
-Value: a single JSON object encoded as a zero-terminated UTF-8 string - -* JSON payload - -```json -{ - "type":"rpm", # this provides a namespace for the package+package-version fields - "os":"fedora", - "osVersion":"33", - "name":"coreutils", - "version":"4711.0815.fc13", - "architecture":"arm32", - "osCpe": "cpe:/o:fedoraproject:fedora:33", # A CPE name for the operating system, `CPE_NAME` from os-release is a good default - "debugInfoUrl": "https://debuginfod.fedoraproject.org/" -} -``` - -The format is a single JSON object, encoded as a zero-terminated `UTF-8` string. -Each name in the object shall be unique as per recommendations of -[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Strings shall -not contain any control character, nor use `\uXXX` escaping. - -When it comes to JSON numbers, this specification assumes that JSON parsers -processing this information are capable of reproducing the full signed 53bit -integer range (i.e. -2⁵³+1…+2⁵³-1) as well as the full 64-bit IEEE floating -point number range losslessly (with the exception of NaN/-inf/+inf, since JSON -cannot encode that), as per recommendations of -[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#page-8). Fields in -these JSON objects are thus permitted to encode numeric values from these -ranges as JSON numbers, and should not use numeric values not covered by these -types and ranges. - -Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes) -are available, and provide macros/helpers to include the note in binaries built -by the package build system. They make use of the new `--package-metadata` flag that -is available in the bfd, gold, mold and lld linkers (versions 2.39, 1.3.0 and 15.0 -respectively). This linker flag takes a JSON payload as parameter. - -## Well-known keys - -The metadata format is intentionally left open, so that vendors can add their own information. -A set of well-known keys is defined here, and hopefully shared among all vendors. - -| Key name | Key description | Example value | -|--------------|--------------------------------------------------------------------------|---------------------------------------| -| type | The packaging type | rpm | -| os | The OS name, typically corresponding to ID in os-release | fedora | -| osVersion | The OS version, typically corresponding to VERSION_ID in os-release | 33 | -| name | The source package name | coreutils | -| version | The source package version | 4711.0815.fc13 | -| architecture | The binary package architecture | arm32 | -| osCpe | A CPE name for the OS, typically corresponding to CPE_NAME in os-release | cpe:/o:fedoraproject:fedora:33 | -| debugInfoUrl | The debuginfod server url, if available | https://debuginfod.fedoraproject.org/ | - -### Displaying package notes - -The raw ELF section can be extracted using `objdump`: -```console -$ objdump -j .note.package -s /usr/bin/ls - -/usr/bin/ls: file format elf64-x86-64 - -Contents of section .note.package: - 03cc 04000000 7c000000 7e1afeca 46444f00 ....|...~...FDO. - 03dc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n - 03ec 616d6522 3a22636f 72657574 696c7322 ame":"coreutils" - 03fc 2c227665 7273696f 6e223a22 392e342d ,"version":"9.4- - 040c 372e6663 3430222c 22617263 68697465 7.fc40","archite - 041c 63747572 65223a22 7838365f 3634222c cture":"x86_64", - 042c 226f7343 7065223a 22637065 3a2f6f3a "osCpe":"cpe:/o: - 043c 6665646f 72617072 6f6a6563 743a6665 fedoraproject:fe - 044c 646f7261 3a343022 7d000000 dora:40"}... -``` - -It is more convenient to use a higher level tool: -```console -$ readelf --notes /usr/bin/ls -... -Displaying notes found in: .note.gnu.build-id - Owner Data size Description - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) - Build ID: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 - -Displaying notes found in: .note.ABI-tag - Owner Data size Description - GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) - OS: Linux, ABI: 3.2.0 - -Displaying notes found in: .note.package - Owner Data size Description - FDO 0x0000007c FDO_PACKAGING_METADATA - Packaging Metadata: {"type":"rpm","name":"coreutils","version":"9.4-7.fc40","architecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:40"} -... - -$ systemd-analyze inspect-elf /usr/bin/ls - path: /usr/bin/ls - elfType: executable -elfArchitecture: AMD x86-64 - - type: rpm - name: coreutils - version: 9.4-7.fc40 - architecture: x86_64 - osCpe: cpe:/o:fedoraproject:fedora:40 - buildId: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 -``` - -If the binary crashes, `systemd-coredump` will display the combined information -from the crashing binary and any shared libraries it links to: - -```console -$ coredumpctl info - PID: 3987823 (ls) - Signal: 11 (SEGV) - Command Line: ls --color=tty -lR / - Executable: /usr/bin/ls -... - Storage: /var/lib/systemd/coredump/core.ls.1000.88dea1b9831c420dbb398f9d2ad9b41e.3987823.1726230641000000.zst (present) - Size on Disk: 194.4K - Package: coreutils/9.4-7.fc40 - build-id: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 - Message: Process 3987823 (ls) of user 1000 dumped core. - - Module /usr/bin/ls from rpm coreutils-9.4-7.fc40.x86_64 - Module libz.so.1 from rpm zlib-ng-2.1.7-1.fc40.x86_64 - Module libcrypto.so.3 from rpm openssl-3.2.2-3.fc40.x86_64 - Module libmount.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64 - Module libcrypt.so.2 from rpm libxcrypt-4.4.36-5.fc40.x86_64 - Module libblkid.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64 - Module libnss_sss.so.2 from rpm sssd-2.9.5-1.fc40.x86_64 - Module libpcre2-8.so.0 from rpm pcre2-10.44-1.fc40.x86_64 - Module libcap.so.2 from rpm libcap-2.69-8.fc40.x86_64 - Module libselinux.so.1 from rpm libselinux-3.6-4.fc40.x86_64 - Stack trace of thread 3987823: - #0 0x00007f19331c3f7e lgetxattr (libc.so.6 + 0x116f7e) - #1 0x00007f19332be4c0 lgetfilecon_raw (libselinux.so.1 + 0x134c0) - #2 0x00007f19332c3bd9 lgetfilecon (libselinux.so.1 + 0x18bd9) - #3 0x000056038273ad55 gobble_file.constprop.0 (/usr/bin/ls + 0x17d55) - #4 0x0000560382733c55 print_dir (/usr/bin/ls + 0x10c55) - #5 0x0000560382727c35 main (/usr/bin/ls + 0x4c35) - #6 0x00007f19330d7088 __libc_start_call_main (libc.so.6 + 0x2a088) - #7 0x00007f19330d714b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a14b) - #8 0x0000560382728f15 _start (/usr/bin/ls + 0x5f15) - ELF object binary architecture: AMD x86-64 -``` - -(This is just a simulation. `ls` is not prone to crashing with a segmentation violation.) diff --git a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md new file mode 100644 index 0000000000..99da8da8d6 --- /dev/null +++ b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md @@ -0,0 +1,197 @@ +--- +title: Package Metadata for ELF Files +category: Interfaces +layout: default +SPDX-License-Identifier: LGPL-2.1-or-later +--- + +# Package Metadata for Core Files + +*Intended audience: hackers working on userspace subsystems that create ELF binaries +or parse ELF core files.* + +## Motivation + +ELF binaries get stamped with a unique, build-time generated hex string identifier called `build-id`, +[which gets embedded as an ELF note called `.note.gnu.build-id`](https://fedoraproject.org/wiki/Releases/FeatureBuildId). +In most cases, this allows a stripped binary to be associated with its debugging information. +It is used, for example, to dynamically fetch DWARF symbols from a debuginfo server, or +to query the local package manager and find out the package metadata or, again, the DWARF +symbols or program sources. + +However, this usage of the `build-id` requires either local metadata, usually set up by +the package manager, or access to a remote server over the network. Both of those might +be unavailable or forbidden. + +Thus it becomes desirable to add additional metadata to a binary at build time, so that +`systemd-coredump` and other services analyzing core files are able to extract said +metadata simply from the core file itself, without external dependencies. + +## Implementation + +This document will attempt to define a common metadata format specification, so that +multiple implementers might use it when building packages, or core file analyzers, and +so on. + +The metadata will be embedded in a single, new, 4-bytes-aligned, allocated, 0-padded, +read-only ELF header section, in a name-value JSON object format. Implementers working on parsing +core files should not assume a specific list of names, but parse anything that is included +in the section, and should look for the note using the `note type`. Implementers working on +build tools should strive to use the same names, for consistency. The most common will be +listed here. When corresponding to the content of os-release, the values should match, again for consistency. + +If available, the metadata should also include the debuginfod server URL that can provide +the original executable, debuginfo and sources, to further facilitate debugging. + +* Section header + +Section: `.note.package`
+`note type`: `0xcafe1a7e`
+Owner: `FDO` (FreeDesktop.org)
+Value: a single JSON object encoded as a zero-terminated UTF-8 string + +* JSON payload + +```json +{ + "type":"rpm", # this provides a namespace for the package+package-version fields + "os":"fedora", + "osVersion":"33", + "name":"coreutils", + "version":"4711.0815.fc13", + "architecture":"arm32", + "osCpe": "cpe:/o:fedoraproject:fedora:33", # A CPE name for the operating system, `CPE_NAME` from os-release is a good default + "debugInfoUrl": "https://debuginfod.fedoraproject.org/" +} +``` + +The format is a single JSON object, encoded as a zero-terminated `UTF-8` string. +Each name in the object shall be unique as per recommendations of +[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Strings shall +not contain any control character, nor use `\uXXX` escaping. + +When it comes to JSON numbers, this specification assumes that JSON parsers +processing this information are capable of reproducing the full signed 53bit +integer range (i.e. -2⁵³+1…+2⁵³-1) as well as the full 64-bit IEEE floating +point number range losslessly (with the exception of NaN/-inf/+inf, since JSON +cannot encode that), as per recommendations of +[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#page-8). Fields in +these JSON objects are thus permitted to encode numeric values from these +ranges as JSON numbers, and should not use numeric values not covered by these +types and ranges. + +Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes) +are available, and provide macros/helpers to include the note in binaries built +by the package build system. They make use of the new `--package-metadata` flag that +is available in the bfd, gold, mold and lld linkers (versions 2.39, 1.3.0 and 15.0 +respectively). This linker flag takes a JSON payload as parameter. + +## Well-known keys + +The metadata format is intentionally left open, so that vendors can add their own information. +A set of well-known keys is defined here, and hopefully shared among all vendors. + +| Key name | Key description | Example value | +|--------------|--------------------------------------------------------------------------|---------------------------------------| +| type | The packaging type | rpm | +| os | The OS name, typically corresponding to ID in os-release | fedora | +| osVersion | The OS version, typically corresponding to VERSION_ID in os-release | 33 | +| name | The source package name | coreutils | +| version | The source package version | 4711.0815.fc13 | +| architecture | The binary package architecture | arm32 | +| osCpe | A CPE name for the OS, typically corresponding to CPE_NAME in os-release | cpe:/o:fedoraproject:fedora:33 | +| debugInfoUrl | The debuginfod server url, if available | https://debuginfod.fedoraproject.org/ | + +### Displaying package notes + +The raw ELF section can be extracted using `objdump`: +```console +$ objdump -j .note.package -s /usr/bin/ls + +/usr/bin/ls: file format elf64-x86-64 + +Contents of section .note.package: + 03cc 04000000 7c000000 7e1afeca 46444f00 ....|...~...FDO. + 03dc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n + 03ec 616d6522 3a22636f 72657574 696c7322 ame":"coreutils" + 03fc 2c227665 7273696f 6e223a22 392e342d ,"version":"9.4- + 040c 372e6663 3430222c 22617263 68697465 7.fc40","archite + 041c 63747572 65223a22 7838365f 3634222c cture":"x86_64", + 042c 226f7343 7065223a 22637065 3a2f6f3a "osCpe":"cpe:/o: + 043c 6665646f 72617072 6f6a6563 743a6665 fedoraproject:fe + 044c 646f7261 3a343022 7d000000 dora:40"}... +``` + +It is more convenient to use a higher level tool: +```console +$ readelf --notes /usr/bin/ls +... +Displaying notes found in: .note.gnu.build-id + Owner Data size Description + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) + Build ID: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 + +Displaying notes found in: .note.ABI-tag + Owner Data size Description + GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag) + OS: Linux, ABI: 3.2.0 + +Displaying notes found in: .note.package + Owner Data size Description + FDO 0x0000007c FDO_PACKAGING_METADATA + Packaging Metadata: {"type":"rpm","name":"coreutils","version":"9.4-7.fc40","architecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:40"} +... + +$ systemd-analyze inspect-elf /usr/bin/ls + path: /usr/bin/ls + elfType: executable +elfArchitecture: AMD x86-64 + + type: rpm + name: coreutils + version: 9.4-7.fc40 + architecture: x86_64 + osCpe: cpe:/o:fedoraproject:fedora:40 + buildId: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 +``` + +If the binary crashes, `systemd-coredump` will display the combined information +from the crashing binary and any shared libraries it links to: + +```console +$ coredumpctl info + PID: 3987823 (ls) + Signal: 11 (SEGV) + Command Line: ls --color=tty -lR / + Executable: /usr/bin/ls +... + Storage: /var/lib/systemd/coredump/core.ls.1000.88dea1b9831c420dbb398f9d2ad9b41e.3987823.1726230641000000.zst (present) + Size on Disk: 194.4K + Package: coreutils/9.4-7.fc40 + build-id: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2 + Message: Process 3987823 (ls) of user 1000 dumped core. + + Module /usr/bin/ls from rpm coreutils-9.4-7.fc40.x86_64 + Module libz.so.1 from rpm zlib-ng-2.1.7-1.fc40.x86_64 + Module libcrypto.so.3 from rpm openssl-3.2.2-3.fc40.x86_64 + Module libmount.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64 + Module libcrypt.so.2 from rpm libxcrypt-4.4.36-5.fc40.x86_64 + Module libblkid.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64 + Module libnss_sss.so.2 from rpm sssd-2.9.5-1.fc40.x86_64 + Module libpcre2-8.so.0 from rpm pcre2-10.44-1.fc40.x86_64 + Module libcap.so.2 from rpm libcap-2.69-8.fc40.x86_64 + Module libselinux.so.1 from rpm libselinux-3.6-4.fc40.x86_64 + Stack trace of thread 3987823: + #0 0x00007f19331c3f7e lgetxattr (libc.so.6 + 0x116f7e) + #1 0x00007f19332be4c0 lgetfilecon_raw (libselinux.so.1 + 0x134c0) + #2 0x00007f19332c3bd9 lgetfilecon (libselinux.so.1 + 0x18bd9) + #3 0x000056038273ad55 gobble_file.constprop.0 (/usr/bin/ls + 0x17d55) + #4 0x0000560382733c55 print_dir (/usr/bin/ls + 0x10c55) + #5 0x0000560382727c35 main (/usr/bin/ls + 0x4c35) + #6 0x00007f19330d7088 __libc_start_call_main (libc.so.6 + 0x2a088) + #7 0x00007f19330d714b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a14b) + #8 0x0000560382728f15 _start (/usr/bin/ls + 0x5f15) + ELF object binary architecture: AMD x86-64 +``` + +(This is just a simulation. `ls` is not prone to crashing with a segmentation violation.) From 03d28cf16f2a3952080f78944cef0ac543a1dc84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Thu, 19 Dec 2024 11:36:03 +0100 Subject: [PATCH 3/6] docs/package-notes: minor corrections Some small grammar cleanups, syntax highligthing, semantic line breaks, missing version in a list. --- docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md index 99da8da8d6..73454ceed1 100644 --- a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md +++ b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md @@ -65,10 +65,11 @@ Value: a single JSON object encoded as a zero-terminated UTF-8 string } ``` -The format is a single JSON object, encoded as a zero-terminated `UTF-8` string. +The format is a single JSON object, +encoded as a zero-terminated `UTF-8` string. Each name in the object shall be unique as per recommendations of -[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Strings shall -not contain any control character, nor use `\uXXX` escaping. +[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). +Strings shall not contain any control characters or use `\uXXX` escaping. When it comes to JSON numbers, this specification assumes that JSON parsers processing this information are capable of reproducing the full signed 53bit @@ -82,9 +83,11 @@ types and ranges. Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes) are available, and provide macros/helpers to include the note in binaries built -by the package build system. They make use of the new `--package-metadata` flag that -is available in the bfd, gold, mold and lld linkers (versions 2.39, 1.3.0 and 15.0 -respectively). This linker flag takes a JSON payload as parameter. +by the package build system. +They make use of the new `--package-metadata=` flag that is available in the +`bfd`, `gold`, `mold`, and `lld` linkers +(versions 2.39, 2.39, 1.3.0, and 15.0 respectively). +This linker flag takes the JSON payload as parameter. ## Well-known keys From 0ab04876f2e82a350d6aa5be5ebb2f4be57c3ef4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Thu, 19 Dec 2024 11:38:03 +0100 Subject: [PATCH 4/6] docs/package-notes: extend motivation and recommendations We described the "how", but not the "why". And the text was a bit dated, still mostly talking about core files. Let's make it more general and describe why a section is used. Add a recommendation/explanation for the debugInfoUrl field. --- docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md index 73454ceed1..1aa8734bce 100644 --- a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md +++ b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md @@ -27,6 +27,14 @@ Thus it becomes desirable to add additional metadata to a binary at build time, `systemd-coredump` and other services analyzing core files are able to extract said metadata simply from the core file itself, without external dependencies. +This metadata is stored as a section in the executable file, +so that it will be loaded into memory along with the text and data of the binary, +and will be preserved in a core dump. +This metadata can also be easily read from the file on disk, +so it can be used to identify provenience of files, +independently of any package management system, +even if the file is renamed or copied. + ## Implementation This document will attempt to define a common metadata format specification, so that @@ -81,6 +89,9 @@ these JSON objects are thus permitted to encode numeric values from these ranges as JSON numbers, and should not use numeric values not covered by these types and ranges. +If available, the metadata should also include the debuginfod server URL that can provide +the original executable, debuginfo and sources, to further facilitate debugging. + Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes) are available, and provide macros/helpers to include the note in binaries built by the package build system. From bafe34ddbe0c90e8addc00a6c3cee4dba5c9426e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Thu, 19 Dec 2024 12:15:59 +0100 Subject: [PATCH 5/6] docs/package-notes: extend description to support PE/COFF files The package notes are nice standard mechanism to figure out the exact name and version of a program or a library stored in a file, without running the code. In the beginning this was designed for the purpose of analyzing core dumps, but we've since started to use it more generically to identify files. The same need to identify PE/COFF files occurs, for example for systemd-boot and other files intended for UEFI. It has also been requested by people working on bootupd (https://github.com/coreos/bootupd). The text is changed to say "Executable" or "ELF or PE/COFF" wherever "ELF" was used previously. The description for PE/COFF files is very brief. AFAIK, we don't have the same "note" mechanism there [1], so just a normal section is used. The alignment rules for PE/UEFI are different and change over time, so no specific alignment is specified. Other requirements for the PE/COFF section are based on what ukify currently does for the .osrel section: 5 .osrel 00000056 000000010132f000 000000010132f000 0002b600 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA The name ".pkgnote" is used as suggested by Daan De Meyer. This corresponds to the ".note.package" name used for ELF. ".package" was also suggested, but I think that is too generic and carries more risk of a conflict with a different use. [1] https://learn.microsoft.com/en-us/windows/win32/debug/pe-format Also zero/0 is replaced by NUL to clarify that a single 0 byte is meant. --- docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md | 49 +++++++++++++------ 1 file changed, 35 insertions(+), 14 deletions(-) diff --git a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md index 1aa8734bce..af564e8e4d 100644 --- a/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md +++ b/docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md @@ -1,14 +1,15 @@ --- -title: Package Metadata for ELF Files +title: Package Metadata for Executable Files category: Interfaces layout: default SPDX-License-Identifier: LGPL-2.1-or-later --- -# Package Metadata for Core Files +# Package Metadata for Executable Files -*Intended audience: hackers working on userspace subsystems that create ELF binaries -or parse ELF core files.* +*Intended audience: hackers working on userspace subsystems that +create or manipulate ELF or PE/COFF binaries +or parse core files.* ## Motivation @@ -41,24 +42,44 @@ This document will attempt to define a common metadata format specification, so multiple implementers might use it when building packages, or core file analyzers, and so on. -The metadata will be embedded in a single, new, 4-bytes-aligned, allocated, 0-padded, -read-only ELF header section, in a name-value JSON object format. Implementers working on parsing -core files should not assume a specific list of names, but parse anything that is included -in the section, and should look for the note using the `note type`. Implementers working on -build tools should strive to use the same names, for consistency. The most common will be -listed here. When corresponding to the content of os-release, the values should match, again for consistency. +Implementers working on parsing the metadata should not assume a specific list of names, +but parse anything that is included in the JSON object. + +Implementers working on build tools should strive to use the same names, for consistency. +The most common will be listed here. +When corresponding to the content of os-release, the values should match, again for consistency. If available, the metadata should also include the debuginfod server URL that can provide the original executable, debuginfo and sources, to further facilitate debugging. -* Section header +### ELF header section + +The metadata will be embedded in a single, 4 byte-aligned, allocated, NUL-padded, +read-only ELF header section, in a name-value JSON object format. +The JSON string is terminated with a NUL +and subsequently padded with NULs to a multiple of four bytes. + +The `note type` must be set during creation and checked when reading. Section: `.note.package`
`note type`: `0xcafe1a7e`
Owner: `FDO` (FreeDesktop.org)
-Value: a single JSON object encoded as a zero-terminated UTF-8 string +Value: a single JSON object encoded as a NUL-terminated UTF-8 string -* JSON payload +### PE/COFF section + +The metadata will be embedded in a single, allocated, NUL-padded, +read-only COFF data section, +in a name-value JSON object format. +The JSON string is terminated with a NUL +and subsequently padded with NULs if appropriate. +The `IMAGE_SCN_CNT_INITIALIZED_DATA` section flag shall be set. +The alignment and padding shall be chosen as appropriate for the use of the PE/COFF file. + +Section: `.pkgnote`
+Value: a single JSON object encoded as a NUL-terminated UTF-8 string + +### JSON payload ```json { @@ -74,7 +95,7 @@ Value: a single JSON object encoded as a zero-terminated UTF-8 string ``` The format is a single JSON object, -encoded as a zero-terminated `UTF-8` string. +encoded as a NUL-terminated `UTF-8` string. Each name in the object shall be unique as per recommendations of [RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Strings shall not contain any control characters or use `\uXXX` escaping. From 06a73d1fb6a8be3f48e727751641d89a5076574f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Thu, 19 Dec 2024 15:32:55 +0100 Subject: [PATCH 6/6] man: update package metadata link and details The variable names are actually "name" and "version". Not sure why the man page had something different. --- man/systemd-analyze.xml | 5 +++-- man/systemd-coredump.xml | 14 ++++++++------ 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/man/systemd-analyze.xml b/man/systemd-analyze.xml index 68d006a1ce..019901d00c 100644 --- a/man/systemd-analyze.xml +++ b/man/systemd-analyze.xml @@ -862,8 +862,9 @@ alias.service:7: Unknown key name 'MysteryKey' in section 'Service', ignoring. This command will load the specified files, and if they are ELF objects (executables, libraries, core files, etc.) it will parse the embedded packaging metadata, if any, and print - it in a table or json format. See the - Packaging Metadata documentation for more information. + it in a table or json format. See the + + Package Metadata for Executable Files document for more information. Print information about a core file as JSON diff --git a/man/systemd-coredump.xml b/man/systemd-coredump.xml index 737b80de9a..a7862f9c0f 100644 --- a/man/systemd-coredump.xml +++ b/man/systemd-coredump.xml @@ -412,12 +412,14 @@ flags: ... COREDUMP_PACKAGE_VERSION= COREDUMP_PACKAGE_JSON= - If the executable contained .package metadata ELF notes, they will be - parsed and attached. The package and packageVersion - of the 'main' ELF module (ie: the executable) will be appended individually. The - JSON-formatted content of all modules will be appended as a single JSON object, each with - the module name as the key. For more information about this metadata format and content, see - the coredump metadata spec. + If the executable contained .package metadata ELF notes, they will be parsed and + attached. The package and version of the 'main' ELF module + (i.e., the executable) will be appended individually. The JSON-formatted content of all modules will + be appended as a single JSON object, each with the module name as the key. For more information about + this metadata format and content, see the + Package Metadata for Executable Files + document. +