diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 410f77dc5e..4d90641d1a 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -7,6 +7,14 @@ on: # Calculate coverage daily at midnight - cron: '0 0 * * *' + pull_request: + branches: + - main + - v[0-9]+-stable + paths: + - .github/workflows/coverage.yml + - test/integration-test-wrapper.py + permissions: contents: read @@ -16,7 +24,7 @@ jobs: steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - - uses: systemd/mkosi@7d45366395f29fdb2b534a850c09d23d29b78fa9 + - uses: systemd/mkosi@ba07d53000b6c560ad0b9f07550aca93c0284e88 # Freeing up disk space with rm -rf can take multiple minutes. Since we don't need the extra free space # immediately, we remove the files in the background. However, we first move them to a different location @@ -49,7 +57,6 @@ jobs: Distribution=arch [Build] - ToolsTree=default ToolsTreeDistribution=arch UseSubvolumes=yes WithTests=no diff --git a/.github/workflows/mkosi.yml b/.github/workflows/mkosi.yml index 3a10d116eb..4d128822fb 100644 --- a/.github/workflows/mkosi.yml +++ b/.github/workflows/mkosi.yml @@ -113,7 +113,7 @@ jobs: steps: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - - uses: systemd/mkosi@7d45366395f29fdb2b534a850c09d23d29b78fa9 + - uses: systemd/mkosi@ba07d53000b6c560ad0b9f07550aca93c0284e88 # Freeing up disk space with rm -rf can take multiple minutes. Since we don't need the extra free space # immediately, we remove the files in the background. However, we first move them to a different location @@ -152,7 +152,6 @@ jobs: [Build] UseSubvolumes=yes - ToolsTree=default ToolsTreeDistribution=fedora ToolsTreeRelease=rawhide diff --git a/docs/HACKING.md b/docs/HACKING.md index cedf35fef0..6326f07916 100644 --- a/docs/HACKING.md +++ b/docs/HACKING.md @@ -7,94 +7,97 @@ SPDX-License-Identifier: LGPL-2.1-or-later # Hacking on systemd -We welcome all contributions to systemd. -If you notice a bug or a missing feature, please feel invited to fix it, and submit your work as a +We welcome all contributions to systemd. If you notice a bug or a missing +feature, please feel invited to fix it, and submit your work as a [GitHub Pull Request (PR)](https://github.com/systemd/systemd/pull/new). -Please make sure to follow our [Coding Style](/CODING_STYLE) when submitting patches. -Also have a look at our [Contribution Guidelines](/CONTRIBUTING). +Please make sure to follow our [Coding Style](/CODING_STYLE) when submitting +patches. Also have a look at our [Contribution Guidelines](/CONTRIBUTING). -When adding new functionality, tests should be added. -For shared functionality (in `src/basic/` and `src/shared/`) unit tests should be sufficient. -The general policy is to keep tests in matching files underneath `src/test/`, -e.g. `src/test/test-path-util.c` contains tests for any functions in `src/basic/path-util.c`. -If adding a new source file, consider adding a matching test executable. -For features at a higher level, tests in `src/test/` are very strongly recommended. -If that is not possible, integration tests in `test/` are encouraged. +When adding new functionality, tests should be added. For shared functionality +(in `src/basic/` and `src/shared/`) unit tests should be sufficient. The general +policy is to keep tests in matching files underneath `src/test/`, e.g. +`src/test/test-path-util.c` contains tests for any functions in +`src/basic/path-util.c`. If adding a new source file, consider adding a matching +test executable. For features at a higher level, tests in `src/test/` are very +strongly recommended. If that is not possible, integration tests in `test/` are +encouraged. Please always test your work before submitting a PR. -Please always test your work before submitting a PR. -For many of the components of systemd testing is straightforward as you can simply compile systemd and run the relevant tool from the build directory. +## Hacking on systemd with mkosi -For some components (most importantly, systemd/PID 1 itself) this is not possible, however. -In order to simplify testing for cases like this we provide a set of `mkosi` config files directly in the source tree. -[mkosi](https://mkosi.systemd.io/) -is a tool for building clean OS images from an upstream distribution in combination with a fresh build of the project in the local working directory. -To make use of this, please install `mkosi` from the [GitHub repository](https://github.com/systemd/mkosi#running-mkosi-from-the-repository). -`mkosi` will build an image for the host distro by default. -First, run `mkosi genkey` to generate a key and certificate to be used for secure boot and verity signing. -After that is done, it is sufficient to type `mkosi` in the systemd project directory to generate a disk image you can boot either in `systemd-nspawn` or in a UEFI-capable VM: +[mkosi](https://mkosi.systemd.io/) is our swiss army knife for hacking on +systemd. It makes sure all necessary dependencies are available to build systemd +and allows building and booting an OS image with the latest systemd installed +for testing purposes. + +First, install `mkosi` from the +[GitHub repository](https://github.com/systemd/mkosi#running-mkosi-from-the-repository). +Note that it's not possible to use your distribution's packaged version of mkosi +as mkosi has to be installed outside of `/usr` for the following steps to work. + +Then, you can build and run systemd executables as follows: ```sh -$ sudo mkosi boot # nspawn still needs sudo for now +$ mkosi -f sandbox meson setup build +$ mkosi -f sandbox ninja -C build +$ mkosi -f sandbox build/systemctl --version ``` -or: +To build and boot an OS image with the latest systemd installed: ```sh -$ mkosi qemu +$ mkosi -f genkey # Generate signing keys once. +$ mkosi -f sandbox ninja -C build mkosi # (re-)build the OS image +$ sudo mkosi boot # Boot the image with systemd-nspawn. +$ mkosi qemu # Boot the image with qemu. ``` -By default, the tools from your host system are used to build the image. -Sometimes we start using mkosi features that rely on functionality in systemd -tools that's not in an official release yet. In that case, you'll need to build -systemd from source on the host and configure mkosi to use the tools from the -systemd build directory. - -To do a local build, most distributions provide very simple and convenient ways -to install most development packages necessary to build systemd: +Putting this all together, here's a series of commands for preparing a patch for +systemd: ```sh -# Fedora -$ sudo dnf builddep systemd -# Debian/Ubuntu -$ sudo apt-get build-dep systemd -# Arch -$ sudo pacman -S devtools -$ pkgctl repo clone --protocol=https systemd +$ git clone https://github.com/systemd/mkosi.git +$ ln -s $PWD/mkosi/bin/mkosi ~/.local/bin/mkosi # Make sure ~/.local/bin is in $PATH. +$ git clone https://github.com/systemd/systemd.git $ cd systemd -$ makepkg -seoc +$ git checkout -b # where BRANCH is the name of the branch +$ $EDITOR src/core/main.c # or wherever you'd like to make your changes +$ mkosi -f sandbox meson setup build # Set up meson +$ mkosi -f genkey # Generate signing keys once. +$ mkosi -f sandbox ninja -C build mkosi # (re-)build the test image +$ mkosi qemu # Boot the image in qemu +$ git add -p # interactively put together your patch +$ git commit # commit it +$ git push -u # where REMOTE is your "fork" on GitHub ``` -After installing the development packages, systemd can be built from source as follows: +And after that, head over to your repo on GitHub and click "Compare & pull +request" -```sh -$ meson setup build -$ ninja -C build -$ meson test -C build -``` +Happy hacking! -To have `mkosi` use the systemd tools from the `build/` directory, add the -following to `mkosi.local.conf`: +The following sections contain advanced topics on how to speed up development or +streamline debugging. Feel free to read them if you're interested but they're +not required to write basic patches. + +## Building the OS image without a tools tree + +By default, `mkosi` will first build a tools tree and use it build the image and +provide the environment for `mkosi sandbox`. To disable the tools tree and use +binaries from your host instead, write the following to `mkosi.local.conf`: ```conf -[Host] -ExtraSearchPaths=build/ +[Build] +ToolsTree= ``` -And if you want `mkosi` to build a tools image and use the tools from there -instead of looking for tools on the host, add the following to -`mkosi.local.conf`: +## Rebuilding systemd without rebuilding the OS image -```conf -[Host] -ToolsTree=default -``` - -Every time you rerun the `mkosi` command a fresh image is built, incorporating -all current changes you made to the project tree. To build the latest changes -and re-install after booting the image, run one of the following commands in -another terminal on your host (choose the right one depending on the -distribution of the container or virtual machine): +Every time the `mkosi` target is built, a fresh image is built. To build the +latest changes and re-install systemd without rebuilding the image, run one of +the following commands in another terminal on your host after booting the image +(choose the right one depending on the distribution of the container or virtual +machine): ```sh mkosi -t none && mkosi ssh dnf upgrade --disablerepo="*" --assumeyes "/work/build/*.rpm" # CentOS/Fedora @@ -107,26 +110,6 @@ and optionally restart the daemon(s) you're working on using `systemctl restart ` or `systemctl daemon-reexec` if you're working on pid1 or `systemctl soft-reboot` to restart everything. -Putting this all together, here's a series of commands for preparing a patch for systemd: - -```sh -$ git clone https://github.com/systemd/mkosi.git -$ ln -s $PWD/mkosi/bin/mkosi /usr/local/bin/mkosi -$ git clone https://github.com/systemd/systemd.git -$ cd systemd -$ git checkout -b # where BRANCH is the name of the branch -$ vim src/core/main.c # or wherever you'd like to make your changes -$ mkosi -f qemu # (re-)build and boot up the test image in qemu -$ mkosi -t none # Build new packages without rebuilding the image -$ git add -p # interactively put together your patch -$ git commit # commit it -$ git push -u # where REMOTE is your "fork" on GitHub -``` - -And after that, head over to your repo on GitHub and click "Compare & pull request" - -Happy hacking! - ## Building distribution packages with mkosi To build distribution packages for a specific distribution and release without @@ -201,67 +184,6 @@ Those are not useful when compiling for distribution and can be disabled by sett See [Testing systemd using sanitizers](/TESTING_WITH_SANITIZERS) for more information on how to build with sanitizers enabled in mkosi. -## Fuzzers - -systemd includes fuzzers in `src/fuzz/` that use libFuzzer and are automatically run by [OSS-Fuzz](https://github.com/google/oss-fuzz) with sanitizers. -To add a fuzz target, create a new `src/fuzz/fuzz-foo.c` file with a `LLVMFuzzerTestOneInput` function and add it to the list in `src/fuzz/meson.build`. - -Whenever possible, a seed corpus and a dictionary should also be added with new fuzz targets. -The dictionary should be named `src/fuzz/fuzz-foo.dict` and the seed corpus should be built and exported as `$OUT/fuzz-foo_seed_corpus.zip` in `tools/oss-fuzz.sh`. - -The fuzzers can be built locally if you have libFuzzer installed by running `tools/oss-fuzz.sh`, or by running: - -```sh -CC=clang CXX=clang++ \ -meson setup build-libfuzz -Dllvm-fuzz=true -Db_sanitize=address,undefined -Db_lundef=false \ --Dc_args='-fno-omit-frame-pointer -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION' -ninja -C build-libfuzz fuzzers -``` - -Each fuzzer then can be then run manually together with a directory containing the initial corpus: - -``` -export UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1 -build-libfuzz/fuzz-varlink-idl test/fuzz/fuzz-varlink-idl/ -``` - -Note: the `halt_on_error=1` UBSan option is especially important, -otherwise the fuzzer won't crash when undefined behavior is triggered. - -You should also confirm that the fuzzers can be built and run using -[the OSS-Fuzz toolchain](https://google.github.io/oss-fuzz/advanced-topics/reproducing/#building-using-docker): - -```sh -path_to_systemd=... - -git clone --depth=1 https://github.com/google/oss-fuzz -cd oss-fuzz - -for sanitizer in address undefined memory; do -for engine in libfuzzer afl honggfuzz; do -./infra/helper.py build_fuzzers --sanitizer "$sanitizer" --engine "$engine" \ ---clean systemd "$path_to_systemd" - -./infra/helper.py check_build --sanitizer "$sanitizer" --engine "$engine" \ --e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd -done -done - -./infra/helper.py build_fuzzers --clean --architecture i386 systemd "$path_to_systemd" -./infra/helper.py check_build --architecture i386 -e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd - -./infra/helper.py build_fuzzers --clean --sanitizer coverage systemd "$path_to_systemd" -./infra/helper.py coverage --no-corpus-download systemd -``` - -If you find a bug that impacts the security of systemd, -please follow the guidance in [CONTRIBUTING.md](/CONTRIBUTING) on how to report a security vulnerability. - -For more details on building fuzzers and integrating with OSS-Fuzz, visit: - -- [Setting up a new project - OSS-Fuzz](https://google.github.io/oss-fuzz/getting-started/new-project-guide/) -- [Tutorials - OSS-Fuzz](https://google.github.io/oss-fuzz/reference/useful-links/#tutorials) - ## Debugging binaries that need to run as root in vscode When trying to debug binaries that need to run as root, diff --git a/mkosi.clangd b/mkosi.clangd index 7cd447c4ee..23251baf63 100755 --- a/mkosi.clangd +++ b/mkosi.clangd @@ -1,5 +1,6 @@ #!/bin/bash # SPDX-License-Identifier: LGPL-2.1-or-later +set -e if command -v flatpak-spawn >/dev/null; then SPAWN=(flatpak-spawn --host) @@ -7,7 +8,7 @@ else SPAWN=() fi -MKOSI_CONFIG="$("${SPAWN[@]}" --host mkosi --json summary | jq -r .Images[-1])" +MKOSI_CONFIG="$("${SPAWN[@]}" mkosi --json summary | jq -r .Images[-1])" DISTRIBUTION="$(jq -r .Distribution <<< "$MKOSI_CONFIG")" RELEASE="$(jq -r .Release <<< "$MKOSI_CONFIG")" ARCH="$(jq -r .Architecture <<< "$MKOSI_CONFIG")" diff --git a/mkosi.conf b/mkosi.conf index 3da4e07a87..ab8a6a1372 100644 --- a/mkosi.conf +++ b/mkosi.conf @@ -29,6 +29,7 @@ RepartDirectories=mkosi.repart OutputDirectory=build/mkosi.output [Build] +ToolsTree=default BuildDirectory=build/mkosi.builddir CacheDirectory=build/mkosi.cache BuildSourcesEphemeral=yes diff --git a/mkosi.conf.d/05-tools/mkosi.conf b/mkosi.conf.d/05-tools/mkosi.conf index 15c336a304..322851b627 100644 --- a/mkosi.conf.d/05-tools/mkosi.conf +++ b/mkosi.conf.d/05-tools/mkosi.conf @@ -3,6 +3,7 @@ [Build] ToolsTreePackages= gcc + gdb gperf lcov llvm diff --git a/mkosi.conf.d/05-tools/mkosi.conf.d/centos.conf b/mkosi.conf.d/05-tools/mkosi.conf.d/centos.conf new file mode 100644 index 0000000000..737fec3a3d --- /dev/null +++ b/mkosi.conf.d/05-tools/mkosi.conf.d/centos.conf @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later + +[Match] +ToolsTreeDistribution=centos + +[Build] +ToolsTreeRepositories=epel,epel-next diff --git a/mkosi.conf.d/05-tools/mkosi.conf.d/opensuse.conf b/mkosi.conf.d/05-tools/mkosi.conf.d/opensuse.conf index c065a52cf3..468f56856f 100644 --- a/mkosi.conf.d/05-tools/mkosi.conf.d/opensuse.conf +++ b/mkosi.conf.d/05-tools/mkosi.conf.d/opensuse.conf @@ -5,6 +5,7 @@ ToolsTreeDistribution=opensuse [Build] ToolsTreePackages= + libz1 gh mypy pkgconfig(blkid) diff --git a/mkosi.conf.d/20-extra-search-path.conf b/mkosi.conf.d/20-extra-search-path.conf new file mode 100644 index 0000000000..b7308f7f21 --- /dev/null +++ b/mkosi.conf.d/20-extra-search-path.conf @@ -0,0 +1,7 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later + +[Match] +PathExists=build/ + +[Build] +ExtraSearchPaths=build/ diff --git a/test/README.md b/test/README.md index fc1c9f136d..7f7a7e658c 100644 --- a/test/README.md +++ b/test/README.md @@ -459,3 +459,72 @@ test unit (be it a static one or a transient one created via systemd-run), with main mount namespace - in that case use `IGNORE_MISSING_COVERAGE=yes` in the test definition (i.e. `TEST-*-NAME/test.sh`), which will skip the post-test check for missing coverage for the respective test. + +## Fuzzers + +systemd includes fuzzers in `src/fuzz/` that use libFuzzer and are automatically +run by [OSS-Fuzz](https://github.com/google/oss-fuzz) with sanitizers. To add a +fuzz target, create a new `src/fuzz/fuzz-foo.c` file with a +`LLVMFuzzerTestOneInput` function and add it to the list in +`src/fuzz/meson.build`. + +Whenever possible, a seed corpus and a dictionary should also be added with new +fuzz targets. The dictionary should be named `src/fuzz/fuzz-foo.dict` and the +seed corpus should be built and exported as `$OUT/fuzz-foo_seed_corpus.zip` in +`tools/oss-fuzz.sh`. + +The fuzzers can be built locally if you have libFuzzer installed by running +`tools/oss-fuzz.sh`, or by running: + +```sh +CC=clang CXX=clang++ \ +meson setup build-libfuzz -Dllvm-fuzz=true -Db_sanitize=address,undefined -Db_lundef=false \ +-Dc_args='-fno-omit-frame-pointer -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION' +ninja -C build-libfuzz fuzzers +``` + +Each fuzzer then can be then run manually together with a directory containing +the initial corpus: + +``` +export UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1 +build-libfuzz/fuzz-varlink-idl test/fuzz/fuzz-varlink-idl/ +``` + +Note: the `halt_on_error=1` UBSan option is especially important, otherwise the +fuzzer won't crash when undefined behavior is triggered. + +You should also confirm that the fuzzers can be built and run using +[the OSS-Fuzz toolchain](https://google.github.io/oss-fuzz/advanced-topics/reproducing/#building-using-docker): + +```sh +path_to_systemd=... + +git clone --depth=1 https://github.com/google/oss-fuzz +cd oss-fuzz + +for sanitizer in address undefined memory; do +for engine in libfuzzer afl honggfuzz; do +./infra/helper.py build_fuzzers --sanitizer "$sanitizer" --engine "$engine" \ +--clean systemd "$path_to_systemd" + +./infra/helper.py check_build --sanitizer "$sanitizer" --engine "$engine" \ +-e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd +done +done + +./infra/helper.py build_fuzzers --clean --architecture i386 systemd "$path_to_systemd" +./infra/helper.py check_build --architecture i386 -e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd + +./infra/helper.py build_fuzzers --clean --sanitizer coverage systemd "$path_to_systemd" +./infra/helper.py coverage --no-corpus-download systemd +``` + +If you find a bug that impacts the security of systemd, please follow the +guidance in [CONTRIBUTING.md](/CONTRIBUTING) on how to report a security +vulnerability. + +For more details on building fuzzers and integrating with OSS-Fuzz, visit: + +- [Setting up a new project - OSS-Fuzz](https://google.github.io/oss-fuzz/getting-started/new-project-guide/) +- [Tutorials - OSS-Fuzz](https://google.github.io/oss-fuzz/reference/useful-links/#tutorials)