SymbX/systemd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Extend elf package metadata to pe/coff files (#35681)

Browse Source
This commit is contained in:
Zbigniew Jędrzejewski-Szmek
2025-01-11 13:12:39 +01:00
committed by GitHub
parent 81a62d9eaf 06a73d1fb6
commit ad7e0fdbf9
7 changed files with 251 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TODO
+3 -3
View File
@@ -701,8 +701,8 @@ Features:
control of the attacker.
* also include packaging metadata (á la
https://systemd.io/ELF_PACKAGE_METADATA/) in our UEFI PE binaries, using the
same JSON format.
https://systemd.io/PACKAGE_METADATA_FOR_EXECUTABLE_FILES/) in our UEFI PE
binaries, using the same JSON format.
* make "bootctl install" + "bootctl update" useful for installing shim too. For
that introduce new dir /usr/lib/systemd/efi/extra/ which we copy mostly 1:1
@@ -2628,7 +2628,7 @@ Features:
* coredump:
- save coredump in Windows/Mozilla minidump format
- when truncating coredumps, also log the full size that the process had, and make a metadata field so we can report truncated coredumps
- add examples for other distros in ELF_PACKAGE_METADATA
- add examples for other distros in PACKAGE_METADATA_FOR_EXECUTABLE_FILES
* support crash reporting operation modes (https://live.gnome.org/GnomeOS/Design/Whiteboards/ProblemReporting)
docs/COREDUMP.md
+2 -2
View File
@@ -68,8 +68,8 @@ specified service unit, and thus can take benefit of regular service resource
management and sandboxing.
The `systemd-coredump` handler will extract a backtrace and
[ELF packaging metadata](/ELF_PACKAGE_METADATA) from any coredumps it
receives and log both.
[ELF packaging metadata](/PACKAGE_METADATA_FOR_EXECUTABLE_FILES)
from any coredumps it receives and log both.
The information about coredumps stored in the journal can be enumerated and queried with the
[`coredumpctl`](https://www.freedesktop.org/software/systemd/man/coredumpctl.html)
tool, for example for directly invoking a debugger such as `gdb` on a collected
docs/COREDUMP_PACKAGE_METADATA.md
+1 -1
View File
@@ -1,4 +1,4 @@
---
layout: forward
target: /ELF_PACKAGE_METADATA
target: /PACKAGE_METADATA_FOR_EXECUTABLE_FILES
---
docs/ELF_PACKAGE_METADATA.md
+2 -197
View File
@@ -1,199 +1,4 @@
---
title: Package Metadata for ELF Files
category: Interfaces
layout: default
SPDX-License-Identifier: LGPL-2.1-or-later
layout: forward
target: /PACKAGE_METADATA_FOR_EXECUTABLE_FILES
---
# Package Metadata for Core Files
*Intended audience: hackers working on userspace subsystems that create ELF binaries
or parse ELF core files.*
## Motivation
ELF binaries get stamped with a unique, build-time generated hex string identifier called `build-id`,
[which gets embedded as an ELF note called `.note.gnu.build-id`](https://fedoraproject.org/wiki/Releases/FeatureBuildId).
In most cases, this allows a stripped binary to be associated with its debugging information.
It is used, for example, to dynamically fetch DWARF symbols from a debuginfo server, or
to query the local package manager and find out the package metadata or, again, the DWARF
symbols or program sources.
However, this usage of the `build-id` requires either local metadata, usually set up by
the package manager, or access to a remote server over the network. Both of those might
be unavailable or forbidden.
Thus it becomes desirable to add additional metadata to a binary at build time, so that
`systemd-coredump` and other services analyzing core files are able to extract said
metadata simply from the core file itself, without external dependencies.
## Implementation
This document will attempt to define a common metadata format specification, so that
multiple implementers might use it when building packages, or core file analyzers, and
so on.
The metadata will be embedded in a single, new, 4-bytes-aligned, allocated, 0-padded,
read-only ELF header section, in a name-value JSON object format. Implementers working on parsing
core files should not assume a specific list of names, but parse anything that is included
in the section, and should look for the note using the `note type`. Implementers working on
build tools should strive to use the same names, for consistency. The most common will be
listed here. When corresponding to the content of os-release, the values should match, again for consistency.
If available, the metadata should also include the debuginfod server URL that can provide
the original executable, debuginfo and sources, to further facilitate debugging.
* Section header
```
SECTION: `.note.package`
note type: `0xcafe1a7e`
Owner: `FDO` (FreeDesktop.org)
Value: a single JSON object encoded as a zero-terminated UTF-8 string
```
* JSON payload
```json
{
"type":"rpm", # this provides a namespace for the package+package-version fields
"os":"fedora",
"osVersion":"33",
"name":"coreutils",
"version":"4711.0815.fc13",
"architecture":"arm32",
"osCpe": "cpe:/o:fedoraproject:fedora:33", # A CPE name for the operating system, `CPE_NAME` from os-release is a good default
"debugInfoUrl": "https://debuginfod.fedoraproject.org/"
}
```
The format is a single JSON object, encoded as a zero-terminated `UTF-8` string.
Each name in the object shall be unique as per recommendations of
[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Strings shall
not contain any control character, nor use `\uXXX` escaping.
When it comes to JSON numbers, this specification assumes that JSON parsers
processing this information are capable of reproducing the full signed 53bit
integer range (i.e. -2⁵³+1…+2⁵³-1) as well as the full 64-bit IEEE floating
point number range losslessly (with the exception of NaN/-inf/+inf, since JSON
cannot encode that), as per recommendations of
[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#page-8). Fields in
these JSON objects are thus permitted to encode numeric values from these
ranges as JSON numbers, and should not use numeric values not covered by these
types and ranges.
Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes)
are available, and provide macros/helpers to include the note in binaries built
by the package build system. They make use of the new `--package-metadata` flag that
is available in the bfd, gold, mold and lld linkers (versions 2.39, 1.3.0 and 15.0
respectively). This linker flag takes a JSON payload as parameter.
## Well-known keys
The metadata format is intentionally left open, so that vendors can add their own information.
A set of well-known keys is defined here, and hopefully shared among all vendors.
| Key name | Key description | Example value |
|--------------|--------------------------------------------------------------------------|---------------------------------------|
| type | The packaging type | rpm |
| os | The OS name, typically corresponding to ID in os-release | fedora |
| osVersion | The OS version, typically corresponding to VERSION_ID in os-release | 33 |
| name | The source package name | coreutils |
| version | The source package version | 4711.0815.fc13 |
| architecture | The binary package architecture | arm32 |
| osCpe | A CPE name for the OS, typically corresponding to CPE_NAME in os-release | cpe:/o:fedoraproject:fedora:33 |
| debugInfoUrl | The debuginfod server url, if available | https://debuginfod.fedoraproject.org/ |
### Displaying package notes
The raw ELF section can be extracted using `objdump`:
```console
$ objdump -j .note.package -s /usr/bin/ls
/usr/bin/ls: file format elf64-x86-64
Contents of section .note.package:
03cc 04000000 7c000000 7e1afeca 46444f00 ....|...~...FDO.
03dc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n
03ec 616d6522 3a22636f 72657574 696c7322 ame":"coreutils"
03fc 2c227665 7273696f 6e223a22 392e342d ,"version":"9.4-
040c 372e6663 3430222c 22617263 68697465 7.fc40","archite
041c 63747572 65223a22 7838365f 3634222c cture":"x86_64",
042c 226f7343 7065223a 22637065 3a2f6f3a "osCpe":"cpe:/o:
043c 6665646f 72617072 6f6a6563 743a6665 fedoraproject:fe
044c 646f7261 3a343022 7d000000 dora:40"}...
```
It is more convenient to use a higher level tool:
```console
$ readelf --notes /usr/bin/ls
...
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.package
Owner Data size Description
FDO 0x0000007c FDO_PACKAGING_METADATA
Packaging Metadata: {"type":"rpm","name":"coreutils","version":"9.4-7.fc40","architecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:40"}
...
$ systemd-analyze inspect-elf /usr/bin/ls
path: /usr/bin/ls
elfType: executable
elfArchitecture: AMD x86-64
type: rpm
name: coreutils
version: 9.4-7.fc40
architecture: x86_64
osCpe: cpe:/o:fedoraproject:fedora:40
buildId: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
```
If the binary crashes, `systemd-coredump` will display the combined information
from the crashing binary and any shared libraries it links to:
```console
$ coredumpctl info
PID: 3987823 (ls)
Signal: 11 (SEGV)
Command Line: ls --color=tty -lR /
Executable: /usr/bin/ls
...
Storage: /var/lib/systemd/coredump/core.ls.1000.88dea1b9831c420dbb398f9d2ad9b41e.3987823.1726230641000000.zst (present)
Size on Disk: 194.4K
Package: coreutils/9.4-7.fc40
build-id: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
Message: Process 3987823 (ls) of user 1000 dumped core.
Module /usr/bin/ls from rpm coreutils-9.4-7.fc40.x86_64
Module libz.so.1 from rpm zlib-ng-2.1.7-1.fc40.x86_64
Module libcrypto.so.3 from rpm openssl-3.2.2-3.fc40.x86_64
Module libmount.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-5.fc40.x86_64
Module libblkid.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64
Module libnss_sss.so.2 from rpm sssd-2.9.5-1.fc40.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.44-1.fc40.x86_64
Module libcap.so.2 from rpm libcap-2.69-8.fc40.x86_64
Module libselinux.so.1 from rpm libselinux-3.6-4.fc40.x86_64
Stack trace of thread 3987823:
#0 0x00007f19331c3f7e lgetxattr (libc.so.6 + 0x116f7e)
#1 0x00007f19332be4c0 lgetfilecon_raw (libselinux.so.1 + 0x134c0)
#2 0x00007f19332c3bd9 lgetfilecon (libselinux.so.1 + 0x18bd9)
#3 0x000056038273ad55 gobble_file.constprop.0 (/usr/bin/ls + 0x17d55)
#4 0x0000560382733c55 print_dir (/usr/bin/ls + 0x10c55)
#5 0x0000560382727c35 main (/usr/bin/ls + 0x4c35)
#6 0x00007f19330d7088 __libc_start_call_main (libc.so.6 + 0x2a088)
#7 0x00007f19330d714b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a14b)
#8 0x0000560382728f15 _start (/usr/bin/ls + 0x5f15)
ELF object binary architecture: AMD x86-64
```
(This is just a simulation. `ls` is not prone to crashing with a segmentation violation.)
docs/PACKAGE_METADATA_FOR_EXECUTABLE_FILES.md
+232
View File
@@ -0,0 +1,232 @@
---
title: Package Metadata for Executable Files
category: Interfaces
layout: default
SPDX-License-Identifier: LGPL-2.1-or-later
---
# Package Metadata for Executable Files
*Intended audience: hackers working on userspace subsystems that
create or manipulate ELF or PE/COFF binaries
or parse core files.*
## Motivation
ELF binaries get stamped with a unique, build-time generated hex string identifier called `build-id`,
[which gets embedded as an ELF note called `.note.gnu.build-id`](https://fedoraproject.org/wiki/Releases/FeatureBuildId).
In most cases, this allows a stripped binary to be associated with its debugging information.
It is used, for example, to dynamically fetch DWARF symbols from a debuginfo server, or
to query the local package manager and find out the package metadata or, again, the DWARF
symbols or program sources.
However, this usage of the `build-id` requires either local metadata, usually set up by
the package manager, or access to a remote server over the network. Both of those might
be unavailable or forbidden.
Thus it becomes desirable to add additional metadata to a binary at build time, so that
`systemd-coredump` and other services analyzing core files are able to extract said
metadata simply from the core file itself, without external dependencies.
This metadata is stored as a section in the executable file,
so that it will be loaded into memory along with the text and data of the binary,
and will be preserved in a core dump.
This metadata can also be easily read from the file on disk,
so it can be used to identify provenience of files,
independently of any package management system,
even if the file is renamed or copied.
## Implementation
This document will attempt to define a common metadata format specification, so that
multiple implementers might use it when building packages, or core file analyzers, and
so on.
Implementers working on parsing the metadata should not assume a specific list of names,
but parse anything that is included in the JSON object.
Implementers working on build tools should strive to use the same names, for consistency.
The most common will be listed here.
When corresponding to the content of os-release, the values should match, again for consistency.
If available, the metadata should also include the debuginfod server URL that can provide
the original executable, debuginfo and sources, to further facilitate debugging.
### ELF header section
The metadata will be embedded in a single, 4 byte-aligned, allocated, NUL-padded,
read-only ELF header section, in a name-value JSON object format.
The JSON string is terminated with a NUL
and subsequently padded with NULs to a multiple of four bytes.
The `note type` must be set during creation and checked when reading.
Section: `.note.package`<br/>
`note type`: `0xcafe1a7e`<br/>
Owner: `FDO` (FreeDesktop.org)<br/>
Value: a single JSON object encoded as a NUL-terminated UTF-8 string
### PE/COFF section
The metadata will be embedded in a single, allocated, NUL-padded,
read-only COFF data section,
in a name-value JSON object format.
The JSON string is terminated with a NUL
and subsequently padded with NULs if appropriate.
The `IMAGE_SCN_CNT_INITIALIZED_DATA` section flag shall be set.
The alignment and padding shall be chosen as appropriate for the use of the PE/COFF file.
Section: `.pkgnote`<br/>
Value: a single JSON object encoded as a NUL-terminated UTF-8 string
### JSON payload
```json
{
"type":"rpm", # this provides a namespace for the package+package-version fields
"os":"fedora",
"osVersion":"33",
"name":"coreutils",
"version":"4711.0815.fc13",
"architecture":"arm32",
"osCpe": "cpe:/o:fedoraproject:fedora:33", # A CPE name for the operating system, `CPE_NAME` from os-release is a good default
"debugInfoUrl": "https://debuginfod.fedoraproject.org/"
}
```
The format is a single JSON object,
encoded as a NUL-terminated `UTF-8` string.
Each name in the object shall be unique as per recommendations of
[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#section-4).
Strings shall not contain any control characters or use `\uXXX` escaping.
When it comes to JSON numbers, this specification assumes that JSON parsers
processing this information are capable of reproducing the full signed 53bit
integer range (i.e. -2⁵³+1…+2⁵³-1) as well as the full 64-bit IEEE floating
point number range losslessly (with the exception of NaN/-inf/+inf, since JSON
cannot encode that), as per recommendations of
[RFC8259](https://datatracker.ietf.org/doc/html/rfc8259#page-8). Fields in
these JSON objects are thus permitted to encode numeric values from these
ranges as JSON numbers, and should not use numeric values not covered by these
types and ranges.
If available, the metadata should also include the debuginfod server URL that can provide
the original executable, debuginfo and sources, to further facilitate debugging.
Reference implementations of [packaging tools for .deb and .rpm](https://github.com/systemd/package-notes)
are available, and provide macros/helpers to include the note in binaries built
by the package build system.
They make use of the new `--package-metadata=` flag that is available in the
`bfd`, `gold`, `mold`, and `lld` linkers
(versions 2.39, 2.39, 1.3.0, and 15.0 respectively).
This linker flag takes the JSON payload as parameter.
## Well-known keys
The metadata format is intentionally left open, so that vendors can add their own information.
A set of well-known keys is defined here, and hopefully shared among all vendors.
| Key name | Key description | Example value |
|--------------|--------------------------------------------------------------------------|---------------------------------------|
| type | The packaging type | rpm |
| os | The OS name, typically corresponding to ID in os-release | fedora |
| osVersion | The OS version, typically corresponding to VERSION_ID in os-release | 33 |
| name | The source package name | coreutils |
| version | The source package version | 4711.0815.fc13 |
| architecture | The binary package architecture | arm32 |
| osCpe | A CPE name for the OS, typically corresponding to CPE_NAME in os-release | cpe:/o:fedoraproject:fedora:33 |
| debugInfoUrl | The debuginfod server url, if available | https://debuginfod.fedoraproject.org/ |
### Displaying package notes
The raw ELF section can be extracted using `objdump`:
```console
$ objdump -j .note.package -s /usr/bin/ls
/usr/bin/ls: file format elf64-x86-64
Contents of section .note.package:
03cc 04000000 7c000000 7e1afeca 46444f00 ....|...~...FDO.
03dc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n
03ec 616d6522 3a22636f 72657574 696c7322 ame":"coreutils"
03fc 2c227665 7273696f 6e223a22 392e342d ,"version":"9.4-
040c 372e6663 3430222c 22617263 68697465 7.fc40","archite
041c 63747572 65223a22 7838365f 3634222c cture":"x86_64",
042c 226f7343 7065223a 22637065 3a2f6f3a "osCpe":"cpe:/o:
043c 6665646f 72617072 6f6a6563 743a6665 fedoraproject:fe
044c 646f7261 3a343022 7d000000 dora:40"}...
```
It is more convenient to use a higher level tool:
```console
$ readelf --notes /usr/bin/ls
...
Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 3.2.0
Displaying notes found in: .note.package
Owner Data size Description
FDO 0x0000007c FDO_PACKAGING_METADATA
Packaging Metadata: {"type":"rpm","name":"coreutils","version":"9.4-7.fc40","architecture":"x86_64","osCpe":"cpe:/o:fedoraproject:fedora:40"}
...
$ systemd-analyze inspect-elf /usr/bin/ls
path: /usr/bin/ls
elfType: executable
elfArchitecture: AMD x86-64
type: rpm
name: coreutils
version: 9.4-7.fc40
architecture: x86_64
osCpe: cpe:/o:fedoraproject:fedora:40
buildId: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
```
If the binary crashes, `systemd-coredump` will display the combined information
from the crashing binary and any shared libraries it links to:
```console
$ coredumpctl info
PID: 3987823 (ls)
Signal: 11 (SEGV)
Command Line: ls --color=tty -lR /
Executable: /usr/bin/ls
...
Storage: /var/lib/systemd/coredump/core.ls.1000.88dea1b9831c420dbb398f9d2ad9b41e.3987823.1726230641000000.zst (present)
Size on Disk: 194.4K
Package: coreutils/9.4-7.fc40
build-id: 40e5a1570a9d97fc48f5c61cfb7690fec0f872b2
Message: Process 3987823 (ls) of user 1000 dumped core.
Module /usr/bin/ls from rpm coreutils-9.4-7.fc40.x86_64
Module libz.so.1 from rpm zlib-ng-2.1.7-1.fc40.x86_64
Module libcrypto.so.3 from rpm openssl-3.2.2-3.fc40.x86_64
Module libmount.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-5.fc40.x86_64
Module libblkid.so.1 from rpm util-linux-2.40.1-1.fc40.x86_64
Module libnss_sss.so.2 from rpm sssd-2.9.5-1.fc40.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.44-1.fc40.x86_64
Module libcap.so.2 from rpm libcap-2.69-8.fc40.x86_64
Module libselinux.so.1 from rpm libselinux-3.6-4.fc40.x86_64
Stack trace of thread 3987823:
#0 0x00007f19331c3f7e lgetxattr (libc.so.6 + 0x116f7e)
#1 0x00007f19332be4c0 lgetfilecon_raw (libselinux.so.1 + 0x134c0)
#2 0x00007f19332c3bd9 lgetfilecon (libselinux.so.1 + 0x18bd9)
#3 0x000056038273ad55 gobble_file.constprop.0 (/usr/bin/ls + 0x17d55)
#4 0x0000560382733c55 print_dir (/usr/bin/ls + 0x10c55)
#5 0x0000560382727c35 main (/usr/bin/ls + 0x4c35)
#6 0x00007f19330d7088 __libc_start_call_main (libc.so.6 + 0x2a088)
#7 0x00007f19330d714b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a14b)
#8 0x0000560382728f15 _start (/usr/bin/ls + 0x5f15)
ELF object binary architecture: AMD x86-64
```
(This is just a simulation. `ls` is not prone to crashing with a segmentation violation.)
man/systemd-analyze.xml
+3 -2
View File
@@ -867,8 +867,9 @@ alias.service:7: Unknown key name 'MysteryKey' in section 'Service', ignoring.
<para>This command will load the specified files, and if they are ELF objects (executables,
libraries, core files, etc.) it will parse the embedded packaging metadata, if any, and print
it in a table or json format. See the <ulink url="https://systemd.io/COREDUMP_PACKAGE_METADATA/">
Packaging Metadata</ulink> documentation for more information.</para>
it in a table or json format. See the
<ulink url="https://systemd.io/PACKAGE_METADATA_FOR_EXECUTABLE_FILES/">
Package Metadata for Executable Files</ulink> document for more information.</para>
<example>
<title>Print information about a core file as JSON</title>
man/systemd-coredump.xml
+8 -6
View File
@@ -412,12 +412,14 @@ flags: ...
<term><varname>COREDUMP_PACKAGE_VERSION=</varname></term>
<term><varname>COREDUMP_PACKAGE_JSON=</varname></term>
<listitem><para>If the executable contained .package metadata ELF notes, they will be
parsed and attached. The <varname>package</varname> and <varname>packageVersion</varname>
of the 'main' ELF module (ie: the executable) will be appended individually. The
JSON-formatted content of all modules will be appended as a single JSON object, each with
the module name as the key. For more information about this metadata format and content, see
<ulink url="https://systemd.io/COREDUMP_PACKAGE_METADATA/">the coredump metadata spec</ulink>.</para>
<listitem><para>If the executable contained .package metadata ELF notes, they will be parsed and
attached. The <varname>package</varname> and <varname>version</varname> of the 'main' ELF module
(i.e., the executable) will be appended individually. The JSON-formatted content of all modules will
be appended as a single JSON object, each with the module name as the key. For more information about
this metadata format and content, see the
<ulink url="https://systemd.io/PACKAGE_METADATA_FOR_EXECUTABLE_FILES/">Package Metadata for Executable Files</ulink>
document.
</para>
<xi:include href="version-info.xml" xpointer="v249"/>
</listitem>