diff --git a/README b/README index 769c5a57a3..80fe3682ff 100644 --- a/README +++ b/README @@ -74,13 +74,18 @@ REQUIREMENTS: ≥ 6.6 for quota support on tmpfs ≥ 6.7 for cgroup2fs memory_hugetlb_accounting option ≥ 6.8 for STATX_MNT_ID_UNIQUE - ≥ 6.9 for pidfs and MPOL_WEIGHTED_INTERLEAVE NUMA policy + ≥ 6.9 for pidfs, PIDFD_THREAD flag for pidfd_open(), and + MPOL_WEIGHTED_INTERLEAVE NUMA policy ≥ 6.10 for fcntl(F_DUPFD_QUERY), unprivileged linkat(AT_EMPTY_PATH), and block device 'partscan' sysfs attribute ≥ 6.12 for AT_HANDLE_MNT_ID_UNIQUE ≥ 6.13 for PIDFD_GET_INFO, {set,remove}xattrat(), and FSCONFIG_SET_FD support for overlayfs layers - ≥ 6.16 for coredump pattern '%F' (pidfd) specifier and SO_PASSRIGHTS + ≥ 6.16 for coredump pattern '%F' (pidfd) specifier and SO_PASSRIGHTS, + PIDFD_INFO_COREDUMP by ioctl(PIDFD_GET_INFO) + ≥ 6.17 for coredump socket pattern ('@@' prefix) + ≥ 6.19 for PIDFD_INFO_SUPPORTED_MASK and PIDFD_INFO_COREDUMP_SIGNAL + ≥ 7.1 for PIDFD_INFO_COREDUMP_CODE ✅ systemd utilizes several new kernel APIs, but will fall back gracefully when unavailable. diff --git a/man/coredump.conf.xml b/man/coredump.conf.xml index 74b025636e..f59c0dacd1 100644 --- a/man/coredump.conf.xml +++ b/man/coredump.conf.xml @@ -47,6 +47,18 @@ and core5 pages for the details. + + On systems where core dumps are processed by the long-running + systemd-coredumpd.service (rather than a fresh + systemd-coredump@.service instance started for every crash, see + systemd-coredump8 for + details on when each applies), these configuration files are only read once, when the service starts. + If systemd-coredumpd.service is running, changes to + coredump.conf or its drop-ins will not take effect until the + service is reloaded: + systemctl reload systemd-coredumpd.service + This causes the running service to reparse its configuration without restarting or interrupting any core + dump currently being processed. @@ -168,9 +180,9 @@ makes sure that old core dumps are removed as soon as the total disk space taken up by core dumps grows beyond this limit (defaults - to 10% of the total disk size, capped at 4GiB, minimum 1MiB). + to 10% of the total disk size, capped at 4GiB, minimum 1MiB). controls how much disk space to keep - free at least (defaults to 15% of the total disk size, limited + free at least (defaults to 15% of the total disk size, limited to a maximum of 4GiB). Note that the disk space used by core dumps might temporarily exceed these limits while core dumps are processed. takes precedence. diff --git a/man/rules/meson.build b/man/rules/meson.build index 35b025c89e..12dc58c6da 100644 --- a/man/rules/meson.build +++ b/man/rules/meson.build @@ -1041,7 +1041,10 @@ manpages = [ ['systemd-clonesetup', '8', ['systemd-clonesetup@.service'], ''], ['systemd-coredump', '8', - ['systemd-coredump.socket', 'systemd-coredump@.service'], + ['systemd-coredump-register.service', + 'systemd-coredump.socket', + 'systemd-coredump@.service', + 'systemd-coredumpd.service'], 'ENABLE_COREDUMP'], ['systemd-creds', '1', [], ''], ['systemd-cryptenroll', '1', [], 'HAVE_LIBCRYPTSETUP'], diff --git a/man/systemd-coredump.xml b/man/systemd-coredump.xml index 4741d25408..6e96a6953f 100644 --- a/man/systemd-coredump.xml +++ b/man/systemd-coredump.xml @@ -20,14 +20,20 @@ systemd-coredump systemd-coredump.socket systemd-coredump@.service + systemd-coredump-register.service + systemd-coredumpd.service Acquire, save and process core dumps /usr/lib/systemd/systemd-coredump /usr/lib/systemd/systemd-coredump - systemd-coredump@.service + /usr/lib/systemd/systemd-coredump + /usr/lib/systemd/systemd-coredump systemd-coredump.socket + systemd-coredump@.service + systemd-coredump-register.service + systemd-coredumpd.service @@ -64,29 +70,193 @@ Handling. - Invocation of <command>systemd-coredump</command> + Overview - The systemd-coredump executable does the actual work. It is invoked twice: - once as the handler by the kernel, and the second time in the - systemd-coredump@.service to actually write the data to the journal and process - and save the core file. + The systemd-coredump executable does the actual work, but is invoked + differently depending on the capabilities of the running kernel. - When the kernel invokes systemd-coredump to handle a core dump, it runs in - privileged mode, and will connect to the socket created by the + When the kernel supports the kernel coredump socket protocol (see "Kernel Requirements" below, + generally kernel v6.19 or newer), systemd-coredumpd.service creates and listens on + a Unix socket, and systemd-coredump-register.service writes the socket path to + /proc/sys/kernel/core_pattern. When a process crashes, the kernel connects to the + socket, and sends the core dump of the process through the socket. Then, + systemd-coredumpd.service transfers the core to the corresponding container if the + crashed process is in a container environment, otherwise saves the core in the local storage and writes + information about the crashed process to journal. + + When the running kernel does not meet those requirements, the executable + systemd-coredump is invoked twice: once as the handler by the kernel, and the second + time in the systemd-coredump@.service to actually write the data to the journal and + process and save the core file. + + When the kernel invokes systemd-coredump to handle a core dump this way, it + runs in privileged mode, and will connect to the socket created by the systemd-coredump.socket unit, which in turn will spawn an unprivileged systemd-coredump@.service instance to process the core dump. Hence systemd-coredump.socket and systemd-coredump@.service are helper units which do the actual processing of core dumps and are subject to normal service management. - It is also possible to invoke systemd-coredump with - option. In this case, systemd-coredump expects a - journal entry in the journal - Journal Export Format - on standard input. The entry should contain a MESSAGE= field and any additional - metadata fields the caller deems reasonable. systemd-coredump will append additional - metadata fields in the same way it does for core dumps received from the kernel. In this mode, no core - dump is stored in the journal. + See "Units" below for details on the individual units involved in either mode. + + + + Command-line Options + + systemd-coredump is normally not invoked directly by the user, but by the + kernel, or by the units described below. It supports the following modes of invocation: + + + + (no option) + + Handles a single core dump. This is the mode the kernel uses to invoke + systemd-coredump directly (via kernel.core_pattern) on + kernels that do not support the kernel coredump socket protocol, as well as the mode used by + systemd-coredump@.service to actually process and save a core dump that was + handed to it via systemd-coredump.socket. + + + + + + systemd-coredump expects a journal entry in the + Journal Export + Format on standard input. The entry should contain a MESSAGE= field and + any additional metadata fields the caller deems reasonable. + systemd-coredump will append additional metadata fields in the same way it does + for core dumps received from the kernel. In this mode, no core dump is stored in the + journal. + + + + + + Runs systemd-coredump as the long-running manager process that + listens on the kernel coredump socket, accepts connections from the kernel, and forks off worker + processes to process each individual core dump. This is the mode used internally by + systemd-coredumpd.service and is not intended to be invoked interactively. + See "Units" below for details. + + + + + + + + Checks whether the running kernel and execution environment fulfil the + requirements for the kernel coredump socket protocol (see "Kernel Requirements" below), and exits + with a non-zero status if they are not met. This is used internally as the + ExecCondition= of systemd-coredumpd.service and + systemd-coredump-register.service, so that those units are skipped, without + being marked as failed, on systems where the feature is unavailable. It is not intended to be + invoked interactively. + + + + + + + + Kernel Requirements for the Coredump Socket + + The kernel coredump socket protocol, used by systemd-coredumpd.service and + systemd-coredump-register.service, requires: + + + A kernel that supports the PIDFD_INFO_COREDUMP (since kernel + v6.16) and PIDFD_INFO_COREDUMP_SIGNAL (since kernel v6.19) flags for the + PIDFD_GET_INFO + ioctl2 + operation on a PID file descriptor (see + pidfd_open2). + Support for these flags is used as an indicator that the kernel is also new enough to support the + request-mode (@@-prefixed) kernel coredump socket + kernel.core_pattern, which was introduced in kernel v6.17. + + systemd-coredump must be running in the initial PID, USER, and + TIME namespaces, i.e. on the host, not inside a container. + + + These conditions are checked with systemd-coredump + before systemd-coredumpd.service and + systemd-coredump-register.service are started. If the requirements are not met, + systemd-coredump instead falls back to the legacy invocation via + systemd-coredump.socket and systemd-coredump@.service. + + + + Units + + The following units are involved in processing core dumps. Which of them are active depends on + whether the kernel coredump socket protocol is available, see "Kernel Requirements" above. + + + + systemd-coredumpd.service + + A long-running manager service, started early at boot on kernels that support the + kernel coredump socket protocol. It creates and listens on the AF_UNIX socket + /run/systemd/coredumpd/kernel (this path is also passed to the kernel via + kernel.core_pattern by systemd-coredump-register.service, + see below). When the kernel connects to this socket to submit a core dump, + systemd-coredumpd.service accepts the connection and forks off a worker + process to receive and process that specific core dump, up to a limit of 8 concurrently running + workers (once this limit is reached, no further connections are accepted until a worker finishes). + Workers that run for longer than 5 minutes are killed automatically. + + The configuration in + coredump.conf5 + is read once when systemd-coredumpd.service starts. Since the service keeps + running afterwards (rather than being invoked anew for every core dump, as + systemd-coredump@.service is), changes to + coredump.conf and its drop-ins are not picked up + automatically. Reload the service to apply configuration changes, see + coredump.conf5 + for details. + + + + + + systemd-coredump-register.service + + A oneshot unit, automatically pulled in and bound to the lifetime of + systemd-coredumpd.service (it is not meant to be started or stopped manually). + Once systemd-coredumpd.service is up, this unit points the kernel at it by + setting kernel.core_pattern to the request-mode (@@-prefixed) + socket path /run/systemd/coredumpd/kernel, and persists this setting to + /run/sysctl.d/60-coredump.conf via + systemd-sysctl8. + When systemd-coredumpd.service is stopped, + systemd-coredump-register.service is stopped along with it and reverts + kernel.core_pattern to its previous value. + + Because /run/sysctl.d/60-coredump.conf takes precedence, per the usual + sysctl.d5 + ordering rules, over the /usr/lib/sysctl.d/50-coredump.conf file described in + "Configuration" below, masking or overriding 50-coredump.conf alone is + not sufficient to install a custom kernel.core_pattern on a + system where systemd-coredumpd.service is active. See "Configuration" below + for how to prevent this. + + + + + + systemd-coredump.socket + systemd-coredump@.service + + Legacy handler units, used on kernels that do not support the kernel coredump + socket protocol (see "Kernel Requirements" above). The kernel invokes + systemd-coredump directly and in privileged mode for each crash; that instance + connects to systemd-coredump.socket, which in turn spawns an unprivileged + systemd-coredump@.service instance to actually process and save the core + dump. Unlike systemd-coredumpd.service, a fresh + systemd-coredump@.service instance is started for every core dump, so it + always reads the current coredump.conf configuration. + + @@ -150,6 +320,21 @@ systemd-sysctl8. + On systems where systemd-coredumpd.service is active (see "Kernel + Requirements" and "Units" above), systemd-coredump-register.service writes its own + kernel.core_pattern setting to /run/sysctl.d/60-coredump.conf + whenever systemd-coredumpd.service is running, and this file takes precedence over + /usr/lib/sysctl.d/50-coredump.conf. As a result, masking or overriding + 50-coredump.conf alone does not override the default + kernel.core_pattern on such systems, since 60-coredump.conf will + still take effect. To set a custom kernel.core_pattern of your own on a system where + systemd-coredumpd.service is available, you must additionally stop (if currently + running) and mask systemd-coredumpd.service: + systemctl mask --now systemd-coredumpd.service + Masking systemd-coredumpd.service also prevents + systemd-coredump-register.service, which is bound to it, from running and writing + 60-coredump.conf. + In order to be used in the mode, an appropriate backtrace handler must be installed on the sender side. For example, in case of python1, this @@ -160,9 +345,14 @@ The behavior of systemd-coredump itself is configured through the configuration file /etc/systemd/coredump.conf and corresponding snippets /etc/systemd/coredump.conf.d/*.conf, see - coredump.conf5. A new - instance of systemd-coredump is invoked upon receiving every core dump. Therefore, changes - in these files will take effect the next time a core dump is received. + coredump.conf5. When + core dumps are processed via systemd-coredump@.service (see "Units" above), a new + instance of systemd-coredump is invoked upon receiving every core dump, and changes in + these files will take effect the next time a core dump is received. When core dumps are instead processed + by the long-running systemd-coredumpd.service, configuration changes require an + explicit reload of that service to take effect; see + coredump.conf5 for + details. Resources used by core dump files are restricted in two ways. Parameters like maximum size of acquired core dumps and files can be set in files /etc/systemd/coredump.conf and snippets mentioned diff --git a/src/basic/process-util.c b/src/basic/process-util.c index ececbf97ac..133acc62b2 100644 --- a/src/basic/process-util.c +++ b/src/basic/process-util.c @@ -350,17 +350,17 @@ int pidref_get_cmdline_strv(const PidRef *pid, ProcessCmdlineFlags flags, char * return 0; } -int pid_is_kernel_thread(pid_t pid) { +int procfs_get_stat_flags(const char *path, unsigned long long *ret) { int r; - if (IN_SET(pid, 0, 1) || pid == getpid_cached()) /* pid 1, and we ourselves certainly aren't a kernel thread */ - return 0; - if (!pid_is_valid(pid)) - return -EINVAL; + assert(path); + assert(ret); + + /* Read the 'flags' field (9th field, task->flags) from the given /proc/[pid]/stat or + * /proc/[pid]/task/[tid]/stat file. */ - const char *p = procfs_file_alloca(pid, "stat"); _cleanup_free_ char *line = NULL; - r = read_one_line_file(p, &line); + r = read_one_line_file(path, &line); if (r == -ENOENT) return -ESRCH; if (r < 0) @@ -397,8 +397,21 @@ int pid_is_kernel_thread(pid_t pid) { return -EINVAL; q[l] = 0; + return safe_atollu(q, ret); +} + +int pid_is_kernel_thread(pid_t pid) { + int r; + + if (IN_SET(pid, 0, 1) || pid == getpid_cached()) /* pid 1, and we ourselves certainly aren't a kernel thread */ + return 0; + if (!pid_is_valid(pid)) + return -EINVAL; + + const char *p = procfs_file_alloca(pid, "stat"); + unsigned long long flags; - r = safe_atollu(q, &flags); + r = procfs_get_stat_flags(p, &flags); if (r < 0) return r; diff --git a/src/basic/process-util.h b/src/basic/process-util.h index 56514c8ed8..1d67ee86a0 100644 --- a/src/basic/process-util.h +++ b/src/basic/process-util.h @@ -72,6 +72,8 @@ int pidref_wait_for_terminate_and_check(const char *name, PidRef *pidref, WaitFl int kill_and_sigcont(pid_t pid, int sig); +int procfs_get_stat_flags(const char *path, unsigned long long *ret); + int pid_is_kernel_thread(pid_t pid); int pidref_is_kernel_thread(const PidRef *pid); diff --git a/src/coredump/coredump-context.c b/src/coredump/coredump-context.c index f7e978bbfc..5138ec7e24 100644 --- a/src/coredump/coredump-context.c +++ b/src/coredump/coredump-context.c @@ -1,5 +1,7 @@ /* SPDX-License-Identifier: LGPL-2.1-or-later */ +#include + #include "sd-login.h" #include "sd-messages.h" @@ -8,6 +10,7 @@ #include "dirent-util.h" #include "fd-util.h" #include "fs-util.h" +#include "hostname-setup.h" #include "hostname-util.h" #include "iovec-wrapper.h" #include "log.h" @@ -16,7 +19,9 @@ #include "parse-util.h" #include "pidfd-util.h" #include "process-util.h" +#include "rlimit-util.h" #include "signal-util.h" +#include "socket-util.h" #include "special.h" #include "string-table.h" #include "string-util.h" @@ -181,7 +186,7 @@ static int get_process_container_parent_cmdline(PidRef *pid, char** ret_cmdline) /* The kernel passes si_signo through core_pattern (%s). Starting with v7.1, * si_code is reported as well via pidfd. */ -static int coredump_context_read_pidfd_info(CoredumpContext *context) { +static int coredump_context_read_code(CoredumpContext *context) { struct pidfd_info info = { .mask = PIDFD_INFO_COREDUMP, }; @@ -190,19 +195,23 @@ static int coredump_context_read_pidfd_info(CoredumpContext *context) { assert(context); assert(pidref_is_set(&context->pidref)); - if (!context->got_pidfd || context->pidref.fd < 0) + if (!context->got_pidfd || context->pidref.fd < 0 || context->got_code >= 0) return 0; r = pidfd_get_info(context->pidref.fd, &info); - if (ERRNO_IS_NEG_NOT_SUPPORTED(r)) - return log_debug_errno(r, "PIDFD_INFO_COREDUMP not supported, ignoring: %m"); - if (r < 0) + if (r < 0) { + context->got_code = false; + + if (ERRNO_IS_NEG_NOT_SUPPORTED(r)) + return log_debug_errno(r, "PIDFD_INFO_COREDUMP not supported, ignoring: %m"); return log_debug_errno(r, "Failed to get pidfd coredump info, ignoring: %m"); + } if (FLAGS_SET(info.mask, PIDFD_INFO_COREDUMP_CODE)) { context->code = (int) info.coredump_code; context->got_code = true; - } + } else + context->got_code = false; return 0; } @@ -246,7 +255,7 @@ int coredump_context_build_iovw(CoredumpContext *context) { (void) iovw_put_string_field(&context->iovw, "COREDUMP_SIGNAL_NAME=SIG", signal_to_string(context->signo)); /* Emit si_code if we learned it from pidfd_info */ - if (context->got_code) + if (context->got_code > 0) (void) iovw_put_string_fieldf(&context->iovw, "COREDUMP_CODE=", "%i", context->code); } @@ -402,7 +411,7 @@ static int coredump_context_parse_from_procfs(CoredumpContext *context) { if (r < 0) log_warning_errno(r, "Failed to get auxv, ignoring: %m"); - (void) coredump_context_read_pidfd_info(context); + (void) coredump_context_read_code(context); r = pidref_verify(&context->pidref); if (r < 0) @@ -645,3 +654,113 @@ int coredump_context_parse_from_argv(CoredumpContext *context, int argc, char ** coredump_context_check_pidns(context); return coredump_context_parse_from_procfs(context); } + +static int get_crashing_tid(pid_t pid, PidRef *ret) { + int r; + + assert(pid > 0); + assert(ret); + + /* Read /proc/[PID]/task/[TID]/stat then find a thread that has the PF_DUMPCORE flag. */ + + const char *p = procfs_file_alloca(pid, "task"); + _cleanup_closedir_ DIR *d = opendir(p); + if (!d) + return errno == ENOENT ? -ESRCH : -errno; + + FOREACH_DIRENT_ALL(de, d, return -errno) { + if (!in_charset(de->d_name, DIGITS)) + continue; + + _cleanup_free_ char *q = path_join(p, de->d_name, "stat"); + if (!q) + return -ENOMEM; + + unsigned long long flags; + r = procfs_get_stat_flags(q, &flags); + if (r == -ENOMEM) + return r; + if (r < 0) { + log_debug_errno(r, "Failed to read '%s', ignoring: %m", q); + continue; + } + + if (!FLAGS_SET(flags, PF_DUMPCORE)) + continue; + + return pidref_set_pidstr_full(ret, de->d_name, PIDFD_THREAD); + } + + return -ESRCH; +} + +int coredump_context_parse_from_peer(CoredumpContext *context) { + int r; + + assert(context); + assert(context->input_fd >= 0); + + r = getpeerpidref(context->input_fd, &context->pidref); + if (r < 0) + return log_error_errno(r, "Failed to get peer pidref: %m"); + + if (context->pidref.fd < 0) + return log_error_errno(SYNTHETIC_ERRNO(ENOMEDIUM), "We do not have pidfd of the crashed process."); + + context->got_pidfd = true; + + struct pidfd_info info = { + .mask = PIDFD_INFO_COREDUMP, + }; + + r = pidfd_get_info(context->pidref.fd, &info); + if (r < 0) + return log_error_errno(r, "ioctl(PIDFD_GET_INFO) failed: %m"); + + if (!FLAGS_SET(info.mask, PIDFD_INFO_COREDUMP | PIDFD_INFO_COREDUMP_SIGNAL)) + return log_error_errno(SYNTHETIC_ERRNO(ENODATA), + "Failed to get coredump information by ioctl(PIDFD_GET_INFO)."); + + if (!FLAGS_SET(info.coredump_mask, PIDFD_COREDUMPED)) + return log_error_errno(SYNTHETIC_ERRNO(EINVAL), + "The process ["PID_FMT"] did not crash.", context->pidref.pid); + + if (FLAGS_SET(info.coredump_mask, PIDFD_COREDUMP_SKIP)) + return log_error_errno(SYNTHETIC_ERRNO(ENODATA), + "Coredump generation for process ["PID_FMT"] is skipped.", context->pidref.pid); + + context->uid = info.ruid; + context->gid = info.rgid; + context->dumpable = FLAGS_SET(info.coredump_mask, PIDFD_COREDUMP_USER) ? SUID_DUMP_USER : SUID_DUMP_SAFE; + context->signo = info.coredump_signal; + + /* This is relatively new (since kernel v7.1), thus optional. */ + if (FLAGS_SET(info.mask, PIDFD_INFO_COREDUMP_CODE)) { + context->code = (int) info.coredump_code; + context->got_code = true; + } else + context->got_code = false; + + r = get_crashing_tid(context->pidref.pid, &context->tidref); + if (r < 0) + log_warning_errno(r, "Failed to get crashing TID of process ["PID_FMT"], ignoring: %m", context->pidref.pid); + + struct rlimit rl; + r = pid_getrlimit(context->pidref.pid, RLIMIT_CORE, &rl); + if (r < 0) + log_warning_errno(r, "Failed to get coredump size limit of process ["PID_FMT"], ignoring: %m", context->pidref.pid); + else + context->rlimit = rl.rlim_cur; + + _cleanup_free_ char *h = NULL; + r = pidref_gethostname_full(&context->pidref, GET_HOSTNAME_ALLOW_LOCALHOST | GET_HOSTNAME_FALLBACK_DEFAULT, &h); + if (r < 0) + log_warning_errno(r, "Failed to get hostname, ignoring: %m"); + else if (!hostname_is_valid(h, /* flags= */ 0)) + log_warning("Obtained an invalid hostname, ignoring: %s", h); + else + context->hostname = TAKE_PTR(h); + + coredump_context_check_pidns(context); + return coredump_context_parse_from_procfs(context); +} diff --git a/src/coredump/coredump-context.h b/src/coredump/coredump-context.h index ee4334dce2..5bb633357e 100644 --- a/src/coredump/coredump-context.h +++ b/src/coredump/coredump-context.h @@ -65,7 +65,7 @@ struct CoredumpContext { size_t auxv_size; /* META_PROC_AUXV */ char *thread_name; /* META_THREAD_NAME */ bool got_pidfd; /* META_ARGV_PIDFD */ - bool got_code; /* META_CODE */ + int got_code; /* META_CODE, tristate */ bool same_pidns; bool forwarded; int input_fd; @@ -79,6 +79,7 @@ struct CoredumpContext { .uid = UID_INVALID, \ .gid = GID_INVALID, \ .tidref = PIDREF_NULL, \ + .got_code = -1, \ .mount_tree_fd = -EBADF, \ .input_fd = -EBADF, \ } @@ -89,3 +90,4 @@ bool coredump_context_is_journald(CoredumpContext *context); int coredump_context_build_iovw(CoredumpContext *context); int coredump_context_parse_iovw(CoredumpContext *context); int coredump_context_parse_from_argv(CoredumpContext *context, int argc, char **argv); +int coredump_context_parse_from_peer(CoredumpContext *context); diff --git a/src/coredump/coredump-manager.c b/src/coredump/coredump-manager.c new file mode 100644 index 0000000000..fd294fd30a --- /dev/null +++ b/src/coredump/coredump-manager.c @@ -0,0 +1,600 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +#include +#include + +#include "acl-util.h" +#include "alloc-util.h" +#include "compress.h" +#include "coredump-config.h" +#include "coredump-manager.h" +#include "coredump-worker.h" +#include "daemon-util.h" +#include "elf-util.h" +#include "errno-util.h" +#include "event-util.h" +#include "fd-util.h" +#include "hashmap.h" +#include "label-util.h" +#include "log.h" +#include "namespace-util.h" +#include "pidfd-util.h" +#include "pidref.h" +#include "process-util.h" +#include "signal-util.h" +#include "socket-util.h" +#include "string-util.h" +#include "strv.h" +#include "time-util.h" +#include "umask-util.h" + +#define COREDUMP_SOCKET_PATH "/run/systemd/coredumpd/kernel" +#define COREDUMP_SOCKET_MAX_WORKERS 8u + +#define WORKER_TIMEOUT_USEC (5 * USEC_PER_MINUTE) + +enum { + /* This should have a higher priority than the others, to prevent accepting new connections. */ + EVENT_PRIORITY_SIGTERM_SIGINT = SD_EVENT_PRIORITY_NORMAL - 4, + /* This should have a higher priority than timer events about killing long running worker processes. */ + EVENT_PRIORITY_WORKER_SIGCHLD = SD_EVENT_PRIORITY_NORMAL - 3, + /* As said in the above, this should have a lower priority than the SIGCHLD event source. */ + EVENT_PRIORITY_WORKER_TIMER = SD_EVENT_PRIORITY_NORMAL - 2, + /* This should have a lower priority than the events for workers. */ + EVENT_PRIORITY_COREDUMP_SOCKET = SD_EVENT_PRIORITY_NORMAL - 1, + /* This should have a lower priority than other event sources. */ + EVENT_PRIORITY_SIGHUP = SD_EVENT_PRIORITY_NORMAL + 1, +}; + +typedef struct Manager Manager; + +typedef struct Worker { + Manager *manager; + sd_event_source *timer_event_source; + sd_event_source *child_event_source; + PidRef pidref; + bool dead; +} Worker; + +struct Manager { + CoredumpConfig config; + sd_event *event; + sd_event_source *coredump_socket_event_source; + int coredump_socket; + Hashmap *workers_by_pidref; + int exit_code; + bool exit; +}; + +static Worker* worker_free(Worker *worker) { + if (!worker) + return NULL; + + if (pidref_is_set(&worker->pidref)) { + if (worker->manager) + hashmap_remove(worker->manager->workers_by_pidref, &worker->pidref); + + if (worker->dead) + pidref_done(&worker->pidref); + else + pidref_done_sigkill_wait(&worker->pidref); + } + + sd_event_source_unref(worker->child_event_source); + sd_event_source_unref(worker->timer_event_source); + + return mfree(worker); +} + +DEFINE_TRIVIAL_CLEANUP_FUNC(Worker*, worker_free); +DEFINE_PRIVATE_HASH_OPS_WITH_VALUE_DESTRUCTOR( + worker_hash_ops, + PidRef, + pidref_hash_func, + pidref_compare_func, + Worker, + worker_free); + +static Worker* worker_new(void) { + Worker *worker = new(Worker, 1); + if (!worker) + return NULL; + + *worker = (Worker) { + .pidref = PIDREF_NULL, + }; + + return worker; +} + +static Manager* manager_free(Manager *manager) { + if (!manager) + return NULL; + + sd_event_source_unref(manager->coredump_socket_event_source); + safe_close(manager->coredump_socket); + hashmap_free(manager->workers_by_pidref); + sd_event_unref(manager->event); + + return mfree(manager); +} + +DEFINE_TRIVIAL_CLEANUP_FUNC(Manager*, manager_free); + +static Manager* manager_new(void) { + Manager *manager = new(Manager, 1); + if (!manager) + return NULL; + + *manager = (Manager) { + .config = COREDUMP_CONFIG_NULL, + .coredump_socket = -EBADF, + }; + + return manager; +} + +static int on_worker_sigchld(sd_event_source *s, const siginfo_t *si, void *userdata) { + _cleanup_(worker_freep) Worker *worker = ASSERT_PTR(userdata); + + assert(si); + + /* We know the worker is dead. Let's avoid calling pidref_done_sigkill_wait() in worker_free(). */ + worker->dead = true; + + switch (si->si_code) { + case CLD_EXITED: + if (si->si_status == 0) + log_debug("Worker ["PID_FMT"] exited.", si->si_pid); + else + log_warning("Worker ["PID_FMT"] exited with code %i.", si->si_pid, si->si_status); + break; + + case CLD_KILLED: + case CLD_DUMPED: + log_warning("Worker ["PID_FMT"] terminated by signal %i (%s).", + si->si_pid, si->si_status, signal_to_string(si->si_status)); + break; + + default: + assert_not_reached(); + } + + return 0; +} + +static int on_worker_timeout(sd_event_source *s, uint64_t usec, void *userdata) { + Worker *worker = ASSERT_PTR(userdata); + + log_warning("Worker ["PID_FMT"] is running longer than %s, killing the worker.", + worker->pidref.pid, FORMAT_TIMESPAN(WORKER_TIMEOUT_USEC, USEC_PER_SEC)); + (void) pidref_kill(&worker->pidref, SIGKILL); + return 0; +} + +static int manager_spawn_worker(Manager *manager, int coredump_fd) { + int r; + + assert(manager); + assert(coredump_fd >= 0); + + _cleanup_(worker_freep) Worker *worker = worker_new(); + if (!worker) + return -ENOMEM; + + /* On socket mode, the kernel does not provide any timestamp of the crash. Let's use the timestamp + * that the socket accept the connection. */ + usec_t timestamp; + r = sd_event_now(manager->event, CLOCK_REALTIME, ×tamp); + if (r < 0) + return r; + + r = pidref_safe_fork_full( + "(coredump-worker)", + /* stdio_fds= */ NULL, + (int[]) { coredump_fd }, 1, + FORK_DEATHSIG_SIGTERM | + FORK_RESET_SIGNALS | + FORK_CLOSE_ALL_FDS | + FORK_REOPEN_LOG | + FORK_LOG, + &worker->pidref); + if (r < 0) + return r; + if (r == 0) { + /* Worker process */ + r = coredump_worker(&manager->config, TAKE_FD(coredump_fd), timestamp); + _exit(r < 0 ? EXIT_FAILURE : EXIT_SUCCESS); + } + + r = event_add_child_pidref(manager->event, &worker->child_event_source, + &worker->pidref, WEXITED, on_worker_sigchld, worker); + if (r < 0) + return r; + + r = sd_event_source_set_priority(worker->child_event_source, EVENT_PRIORITY_WORKER_SIGCHLD); + if (r < 0) + return r; + + (void) sd_event_source_set_description(worker->child_event_source, "worker-child-event"); + + r = sd_event_add_time_relative(manager->event, &worker->timer_event_source, + CLOCK_MONOTONIC, WORKER_TIMEOUT_USEC, USEC_PER_SEC, + on_worker_timeout, worker); + if (r < 0) + return r; + + r = sd_event_source_set_priority(worker->timer_event_source, EVENT_PRIORITY_WORKER_TIMER); + if (r < 0) + return r; + + (void) sd_event_source_set_description(worker->timer_event_source, "worker-timeout"); + + r = hashmap_ensure_put(&manager->workers_by_pidref, &worker_hash_ops, &worker->pidref, worker); + if (r < 0) + return r; + + worker->manager = manager; + + TAKE_PTR(worker); + return 0; +} + +static int on_connect(sd_event_source *s, int fd, uint32_t revents, void *userdata) { + Manager *manager = ASSERT_PTR(userdata); + int r; + + assert(fd >= 0); + assert(!manager->exit); + + _cleanup_close_ int coredump_fd = accept4(fd, /* addr= */ NULL, /* addrlen= */ NULL, SOCK_CLOEXEC | SOCK_NONBLOCK); + if (coredump_fd < 0) { + if (!ERRNO_IS_ACCEPT_AGAIN(errno)) + log_warning_errno(errno, "Failed to accept coredump socket connection, ignoring: %m"); + + return 0; + } + + r = manager_spawn_worker(manager, coredump_fd); + if (r < 0) + log_warning_errno(r, "Failed to spawn worker process, ignoring: %m"); + + if (hashmap_size(manager->workers_by_pidref) >= COREDUMP_SOCKET_MAX_WORKERS) { + r = sd_event_source_set_enabled(s, SD_EVENT_OFF); + if (r < 0) + log_warning_errno(r, "Failed to stop listening on the kernel coredump socket, ignoring: %m"); + } + + return 0; +} + +static int manager_exit(Manager *manager, int exit_code) { + assert(manager); + + if (manager->exit) + return 0; + + manager->exit_code = exit_code; + manager->exit = true; + + (void) sd_notify(/* unset_environment= */ false, NOTIFY_STOPPING_MESSAGE); + + /* Do not accept any new connections. */ + manager->coredump_socket_event_source = sd_event_source_disable_unref(manager->coredump_socket_event_source); + manager->coredump_socket = safe_close(manager->coredump_socket); + + return 0; +} + +static int on_post(sd_event_source *s, void *userdata) { + Manager *manager = ASSERT_PTR(userdata); + int r; + + if (!manager->exit && + hashmap_size(manager->workers_by_pidref) < COREDUMP_SOCKET_MAX_WORKERS) { + r = sd_event_source_set_enabled(manager->coredump_socket_event_source, SD_EVENT_ON); + if (r < 0) { + log_error_errno(r, "Failed to re-enable listening on the kernel coredump socket, terminating: %m"); + (void) manager_exit(manager, r); + } + } + + if (manager->exit) { + if (!hashmap_isempty(manager->workers_by_pidref)) + return 0; /* There still exist workers. */ + + return sd_event_exit(manager->event, manager->exit_code); + } + + return 0; +} + +static int on_sigterm(sd_event_source *s, const struct signalfd_siginfo *si, void *userdata) { + return manager_exit(ASSERT_PTR(userdata), /* exit_code= */ 0); +} + +static int on_sighup(sd_event_source *s, const struct signalfd_siginfo *si, void *userdata) { + Manager *manager = ASSERT_PTR(userdata); + int r; + + if (manager->exit) + return 0; + + (void) notify_reloading(); + + manager->config = COREDUMP_CONFIG_NULL; + (void) coredump_parse_config(&manager->config); + + r = sd_notify(/* unset_environment= */ false, NOTIFY_READY_MESSAGE); + if (r < 0) + log_warning_errno(r, "Failed to send readiness notification, ignoring: %m"); + + return 0; +} + +static int manager_setup_signal( + Manager *manager, + sd_event *event, + int signal, + sd_event_signal_handler_t handler, + int64_t priority, + const char *description) { + + int r; + + assert(manager); + assert(event); + + _cleanup_(sd_event_source_unrefp) sd_event_source *s = NULL; + r = sd_event_add_signal(event, &s, signal | SD_EVENT_SIGNAL_PROCMASK, handler, manager); + if (r < 0) + return r; + + r = sd_event_source_set_priority(s, priority); + if (r < 0) + return r; + + (void) sd_event_source_set_description(s, description); + + r = sd_event_source_set_floating(s, true); + if (r < 0) + return r; + + return 0; +} + +static int manager_init_event(Manager *manager) { + int r; + + assert(manager); + + /* block SIGCHLD for listening child events. */ + assert_se(sigprocmask_many(SIG_BLOCK, /* ret_old_mask= */ NULL, SIGCHLD) >= 0); + + _cleanup_(sd_event_unrefp) sd_event *event = NULL; + r = sd_event_default(&event); + if (r < 0) + return log_error_errno(r, "Failed to allocate event loop: %m"); + + r = sd_event_set_watchdog(event, true); + if (r < 0) + return log_error_errno(r, "Failed to enable watchdog: %m"); + + r = manager_setup_signal(manager, event, SIGINT, on_sigterm, + EVENT_PRIORITY_SIGTERM_SIGINT, "sigint-event-source"); + if (r < 0) + return log_error_errno(r, "Failed to create SIGINT event source: %m"); + + r = manager_setup_signal(manager, event, SIGTERM, on_sigterm, + EVENT_PRIORITY_SIGTERM_SIGINT, "sigterm-event-source"); + if (r < 0) + return log_error_errno(r, "Failed to create SIGTERM event source: %m"); + + r = manager_setup_signal(manager, event, SIGHUP, on_sighup, + EVENT_PRIORITY_SIGHUP, "sighup-event-source"); + if (r < 0) + return log_error_errno(r, "Failed to create SIGHUP event source: %m"); + + r = sd_event_add_post(event, /* ret= */ NULL, on_post, manager); + if (r < 0) + return log_error_errno(r, "Failed to create post event source: %m"); + + _cleanup_(sd_event_source_unrefp) sd_event_source *s = NULL; + r = sd_event_add_io(event, &s, manager->coredump_socket, EPOLLIN, on_connect, manager); + if (r < 0) + return log_error_errno(r, "Failed to allocate IO event source: %m"); + + (void) sd_event_source_set_description(s, "coredump-socket-event"); + + r = sd_event_source_set_priority(s, EVENT_PRIORITY_COREDUMP_SOCKET); + if (r < 0) + return log_error_errno(r, "Failed to set priority of IO event source: %m"); + + manager->coredump_socket_event_source = TAKE_PTR(s); + manager->event = TAKE_PTR(event); + return 0; +} + +static int manager_open_coredump_socket(Manager *manager) { + int r; + + assert(manager); + assert(manager->coredump_socket < 0); + + _cleanup_close_ int fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0); + if (fd < 0) + return log_error_errno(errno, "Failed to create AF_UNIX socket: %m"); + + union sockaddr_union sa; + r = sockaddr_un_set_path(&sa.un, COREDUMP_SOCKET_PATH); + if (r < 0) + return log_error_errno(r, "Failed to set path for AF_UNIX socket: %m"); + socklen_t sa_len = r; + + (void) sockaddr_un_unlink(&sa.un); + + WITH_UMASK(0077) + r = RET_NERRNO(bind(fd, &sa.sa, sa_len)); + if (r < 0) + return log_error_errno(r, "Failed to bind AF_UNIX socket at %s: %m", sa.un.sun_path); + + if (listen(fd, SOMAXCONN_DELUXE) < 0) + return log_error_errno(errno, "Failed to listen AF_UNIX socket at %s: %m", sa.un.sun_path); + + manager->coredump_socket = TAKE_FD(fd); + return 0; +} + +static int manager_push_coredump_socket(Manager *manager) { + int r; + + assert(manager); + assert(manager->coredump_socket >= 0); + + r = notify_push_fd(manager->coredump_socket, "coredump-socket"); + if (r < 0) + return log_warning_errno(r, "Failed to push coredump socket to service manager, ignoring: %m"); + + log_debug("Pushed coredump socket to service manager."); + return 0; +} + +static int manager_set_coredump_socket(Manager *manager, int fd) { + int r; + + assert(manager); + assert(fd >= 0); + + /* This takes passed fd on success. */ + + if (manager->coredump_socket >= 0) + return log_warning_errno(SYNTHETIC_ERRNO(EALREADY), + "Received multiple coredump socket (%i), ignoring.", fd); + + r = sd_is_socket_unix(fd, SOCK_STREAM, /* listening= */ true, COREDUMP_SOCKET_PATH, /* length= */ 0); + if (r < 0) + return log_warning_errno(r, "Failed to check if fd (%i) is a valid unix socket, ignoring: %m", fd); + if (r == 0) + return log_warning_errno(SYNTHETIC_ERRNO(EINVAL), + "Received invalid coredump socket (%i), ignoring.", fd); + + (void) fd_nonblock(fd, true); + + manager->coredump_socket = fd; + return 0; +} + +static int manager_listen_fds(Manager *manager) { + int r; + + assert(manager); + + _cleanup_strv_free_ char **names = NULL; + int n = sd_listen_fds_with_names(/* unset_environment= */ false, &names); + if (n < 0) + return log_error_errno(n, "Failed to determine the number of file descriptors: %m"); + + for (int i = 0; i < n; i++) { + int fd = SD_LISTEN_FDS_START + i; + + if (streq_ptr(names[i], "coredump-socket")) + r = manager_set_coredump_socket(manager, fd); + else + r = log_warning_errno(SYNTHETIC_ERRNO(EINVAL), + "Received unexpected fd (%i: %s), ignoring.", fd, names[i]); + if (r < 0) + close_and_notify_warn(fd, names[i]); + } + + return 0; +} + +int coredump_manager_check_requirements(int log_level) { + int r; + + /* This checks the following flags are supported by the running kernel: + * PIDFD_INFO_COREDUMP : 1d8db6fd698de1f73b1a7d72aea578fdd18d9a87 (v6.16), + * PIDFD_INFO_COREDUMP_SIGNAL : 036375522be8425874e9e0f907c7127e315c7a52 (v6.19). + * + * These flags are required for using the kernel coredump socket feature. Note that the + * request mode coredump socket pattern (@@ prefixed) is supported since kernel v6.17. Old + * kernels do not refuse the new core patterns (moreover, any strings are accepted), hence we + * need to check kernel version in some ways other than reading/writing core patterns. This + * method can be also used for checking that. If the required flags are supported, we assume + * the kernel is new enough, and the kernel coredump socket is also supported. */ + r = pidfd_info_mask_is_supported(PIDFD_INFO_COREDUMP | PIDFD_INFO_COREDUMP_SIGNAL); + if (IN_SET(r, 0, -EOPNOTSUPP)) + return log_full_errno(log_level, SYNTHETIC_ERRNO(EOPNOTSUPP), + "ioctl(PIDFD_GET_INFO) does not support PIDFD_INFO_COREDUMP and/or PIDFD_INFO_COREDUMP_SIGNAL."); + if (r < 0) + return log_error_errno(r, "Failed to check if PIDFD_INFO_COREDUMP and PIDFD_INFO_COREDUMP_SIGNAL flags are supported by ioctl(PIDFD_GET_INFO): %m"); + + log_debug("The kernel supports both PIDFD_INFO_COREDUMP and PIDFD_INFO_COREDUMP_SIGNAL, assuming the kernel coredump socket is also supported."); + + /* Let's check if we are in the initial PID, USER, TIME namespace. */ + NamespaceType type; + FOREACH_ARGUMENT(type, NAMESPACE_PID, NAMESPACE_USER, NAMESPACE_TIME) { + r = namespace_is_init(type); + if (r < 0) + return log_error_errno(r, "Failed to check if we are in the initial %s namespace: %m", + namespace_info[type].proc_name); + if (r == 0) + return log_full_errno(log_level, SYNTHETIC_ERRNO(EOPNOTSUPP), + "Running in a non-initial %s namespace.", + namespace_info[type].proc_name); + } + + return 0; +} + +int coredump_manager(void) { + int r; + + log_setup(); + + r = coredump_manager_check_requirements(LOG_ERR); + if (r < 0) + return r; + + r = mac_init(); + if (r < 0) + return r; + + /* Load some shared libraries before we fork any workers */ + (void) dlopen_compress_default(LOG_DEBUG); + (void) dlopen_dw(LOG_DEBUG); + (void) dlopen_elf(LOG_DEBUG); + (void) dlopen_libacl(LOG_DEBUG); + + _cleanup_(manager_freep) Manager *manager = manager_new(); + if (!manager) + return log_oom(); + + /* Ignore all parse errors. */ + (void) coredump_parse_config(&manager->config); + + r = manager_listen_fds(manager); + if (r < 0) + return r; + + if (manager->coredump_socket < 0) { + r = manager_open_coredump_socket(manager); + if (r < 0) + return r; + + (void) manager_push_coredump_socket(manager); + } + + r = manager_init_event(manager); + if (r < 0) + return r; + + _unused_ _cleanup_(notify_on_cleanup) const char *notify_message = + notify_start(NOTIFY_READY_MESSAGE, NOTIFY_STOPPING_MESSAGE); + + r = sd_event_loop(manager->event); + if (r < 0) + return log_error_errno(r, "Event loop failed: %m"); + + return 0; +} diff --git a/src/coredump/coredump-manager.h b/src/coredump/coredump-manager.h new file mode 100644 index 0000000000..3af742e1b0 --- /dev/null +++ b/src/coredump/coredump-manager.h @@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +#pragma once + +int coredump_manager_check_requirements(int log_level); +int coredump_manager(void); diff --git a/src/coredump/coredump-worker.c b/src/coredump/coredump-worker.c new file mode 100644 index 0000000000..d7dc78655d --- /dev/null +++ b/src/coredump/coredump-worker.c @@ -0,0 +1,289 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ + +#include + +#include "sd-event.h" + +#include "coredump-context.h" +#include "coredump-send.h" +#include "coredump-submit.h" +#include "coredump-worker.h" +#include "errno-util.h" +#include "fd-util.h" +#include "format-util.h" +#include "log.h" +#include "process-util.h" +#include "signal-util.h" +#include "socket-util.h" + +#define COREDUMP_REQ_SIZE_MAX 4096u + +typedef enum CoredumpHandshakeState { + COREDUMP_HANDSHAKE_WAITING_REQUEST, + COREDUMP_HANDSHAKE_SENDING_ACK, + COREDUMP_HANDSHAKE_WAITING_MARKER, + COREDUMP_HANDSHAKE_DONE, + _COREDUMP_HANDSHAKE_STATE_MAX, + _COREDUMP_HANDSHAKE_STATE_INVALID = -EINVAL, +} CoredumpHandshakeState; + +typedef struct HandshakeContext { + CoredumpHandshakeState state; + struct coredump_req req; +} HandshakeContext; + +static int handshake_process_request(HandshakeContext *hc, sd_event_source *s, int fd, uint32_t revents) { + int r; + + assert(hc); + assert(hc->state == COREDUMP_HANDSHAKE_WAITING_REQUEST); + assert(s); + assert(fd >= 0); + assert(FLAGS_SET(revents, EPOLLIN)); + + ssize_t n = next_datagram_size_fd(fd); + if (n < 0) { + if (ERRNO_IS_NEG_TRANSIENT(n)) + return 0; + return log_debug_errno(n, "Failed to determine coredump request size: %m"); + } + + /* Verify the acquired request size. */ + if (n < COREDUMP_REQ_SIZE_VER0) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Acquired coredump request size is too small (%zi < %i).", + n, COREDUMP_REQ_SIZE_VER0); + if ((size_t) n > COREDUMP_REQ_SIZE_MAX) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Acquired coredump request size is too large (%zi > %u).", + n, COREDUMP_REQ_SIZE_MAX); + + union coredump_req_union { + struct coredump_req req; + uint8_t buf[COREDUMP_REQ_SIZE_MAX]; + } req = {}; + + n = recv(fd, &req, n, /* flags= */ 0); + if (n < 0) { + if (ERRNO_IS_TRANSIENT(errno)) + return 0; + return log_debug_errno(errno, "Failed to receive coredump request: %m"); + } + + assert((size_t) n <= sizeof(req)); + + /* Verify the received coredump request. */ + if (n < COREDUMP_REQ_SIZE_VER0) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Received coredump request size is too small (%zi < %i).", + n, COREDUMP_REQ_SIZE_VER0); + if ((size_t) n != req.req.size) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Received coredump request size does not match with the size specified in the request (%zi != %"PRIu32").", + n, req.req.size); + if (req.req.size_ack < COREDUMP_ACK_SIZE_VER0) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Received coredump request with too small ack size (%"PRIu32" < %i).", + req.req.size_ack, COREDUMP_ACK_SIZE_VER0); + if (!FLAGS_SET(req.req.mask, COREDUMP_KERNEL | COREDUMP_USERSPACE | COREDUMP_REJECT | COREDUMP_WAIT)) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Received coredump request with insufficient flags (%"PRIx64").", + (uint64_t) req.req.mask); + + hc->req = req.req; + + log_debug("Received coredump request, sending coredump ack."); + hc->state = COREDUMP_HANDSHAKE_SENDING_ACK; + + r = sd_event_source_set_io_events(s, EPOLLOUT); + if (r < 0) + return log_debug_errno(r, "Failed to set IO events: %m"); + + return 0; +} + +static int handshake_send_ack(HandshakeContext *hc, sd_event_source *s, int fd, uint32_t revents) { + int r; + + assert(hc); + assert(hc->state == COREDUMP_HANDSHAKE_SENDING_ACK); + assert(s); + assert(fd >= 0); + assert(FLAGS_SET(revents, EPOLLOUT)); + + struct coredump_ack ack = { + /* send the full coredump_ack structure if we can, but never more than what the kernel actually supports */ + .size = MIN(sizeof(struct coredump_ack), hc->req.size_ack), + .mask = COREDUMP_KERNEL | COREDUMP_WAIT, + }; + + ssize_t n = send(fd, &ack, ack.size, MSG_NOSIGNAL); + if (n < 0) { + if (ERRNO_IS_TRANSIENT(errno)) + return 0; + return log_debug_errno(errno, "Failed to send coredump ack: %m"); + } + if ((size_t) n != ack.size) + return log_debug_errno(SYNTHETIC_ERRNO(ESTALE), + "Sent size does not match with the size of coredump ack (%zi != %"PRIu32").", + n, ack.size); + + log_debug("Sent coredump ack, waiting for marker."); + hc->state = COREDUMP_HANDSHAKE_WAITING_MARKER; + + r = sd_event_source_set_io_events(s, EPOLLIN); + if (r < 0) + return log_debug_errno(r, "Failed to set IO events: %m"); + + return 0; +} + +static int handshake_process_marker(HandshakeContext *hc, sd_event_source *s, int fd, uint32_t revents) { + assert(hc); + assert(hc->state == COREDUMP_HANDSHAKE_WAITING_MARKER); + assert(s); + assert(fd >= 0); + assert(FLAGS_SET(revents, EPOLLIN)); + + enum coredump_mark mark; + ssize_t n = recv(fd, &mark, sizeof(mark), /* flags= */ 0); + if (n < 0) { + if (ERRNO_IS_TRANSIENT(errno)) + return 0; + return log_debug_errno(errno, "Failed to receive marker: %m"); + } + if ((size_t) n != sizeof(mark)) + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), "Received marker with invalid size (%zi).", n); + + switch (mark) { + case COREDUMP_MARK_REQACK: + log_debug("Sent coredump ack message is accepted, reading coredump data."); + hc->state = COREDUMP_HANDSHAKE_DONE; + return 1; + case COREDUMP_MARK_MINSIZE: + return log_debug_errno(SYNTHETIC_ERRNO(ENOBUFS), + "Sent coredump ack message is refused as its size is too small."); + case COREDUMP_MARK_MAXSIZE: + return log_debug_errno(SYNTHETIC_ERRNO(EMSGSIZE), + "Sent coredump ack message is refused as its size is too large."); + case COREDUMP_MARK_UNSUPPORTED: + return log_debug_errno(SYNTHETIC_ERRNO(EOPNOTSUPP), + "Sent coredump ack message is refused as it contains unsupported flags."); + case COREDUMP_MARK_CONFLICTING: + return log_debug_errno(SYNTHETIC_ERRNO(EINVAL), + "Sent coredump ack message is refused as it contains conflicting flags."); + default: + return log_debug_errno(SYNTHETIC_ERRNO(EBADMSG), + "Sent coredump ack message is refused with unknown reason (%u).", mark); + } +} + +static int on_handshake_io(sd_event_source *s, int fd, uint32_t revents, void *userdata) { + HandshakeContext *hc = ASSERT_PTR(userdata); + int r; + + assert(s); + assert(fd >= 0); + + if (revents & (EPOLLHUP|EPOLLERR)) { + log_debug("Received EPOLLHUP and/or EPOLLERR."); + return sd_event_exit(sd_event_source_get_event(s), -EIO); + } + + switch (hc->state) { + case COREDUMP_HANDSHAKE_WAITING_REQUEST: + r = handshake_process_request(hc, s, fd, revents); + break; + case COREDUMP_HANDSHAKE_SENDING_ACK: + r = handshake_send_ack(hc, s, fd, revents); + break; + case COREDUMP_HANDSHAKE_WAITING_MARKER: + r = handshake_process_marker(hc, s, fd, revents); + break; + default: + assert_not_reached(); + } + if (r != 0) + return sd_event_exit(sd_event_source_get_event(s), r < 0 ? r : 0); + + return 0; +} + +static int coredump_handshake(CoredumpContext *context) { + int r; + + assert(context); + assert(context->input_fd >= 0); + + _cleanup_(sd_event_unrefp) sd_event *e = NULL; + r = sd_event_new(&e); + if (r < 0) + return log_error_errno(r, "Failed to allocate sd-event object: %m"); + + HandshakeContext hc = {}; + r = sd_event_add_io(e, /* ret= */ NULL, context->input_fd, EPOLLIN, on_handshake_io, &hc); + if (r < 0) + return log_error_errno(r, "Failed to add IO event source for kernel coredump socket: %m"); + + r = sd_event_set_signal_exit(e, true); + if (r < 0) + return log_error_errno(r, "Failed to enable signal event sources: %m"); + + log_debug("Processing coredump request."); + + r = sd_event_loop(e); + if (r < 0) + return log_error_errno(r, "Failed to process coredump request: %m"); + + if (hc.state != COREDUMP_HANDSHAKE_DONE) /* Maybe received SIGTERM/SIGINT. */ + return log_error_errno(SYNTHETIC_ERRNO(EIO), "Processing coredump request is cancelled."); + + return 0; +} + +int coredump_worker(const CoredumpConfig *config, int coredump_fd, usec_t timestamp) { + int r; + + assert(config); + assert(coredump_fd >= 0); + + /* This invalidates the input file descriptor even on failure. */ + + LogTarget saved_target = _LOG_TARGET_INVALID; + if (!log_on_console()) { + saved_target = log_get_target(); + log_set_target_and_open(LOG_TARGET_KMSG); + } + + /* Set higher OOM score, we only protect the manager process. */ + r = set_oom_score_adjust(500); + if (r < 0) + log_debug_errno(r, "Failed to reset OOM score, ignoring: %m"); + + _cleanup_(coredump_context_done) CoredumpContext context = COREDUMP_CONTEXT_NULL; + context.input_fd = TAKE_FD(coredump_fd); + context.timestamp = timestamp; + + r = coredump_context_parse_from_peer(&context); + if (r < 0) + return r; + + if (!coredump_context_is_journald(&context) && saved_target >= 0) + log_set_target_and_open(saved_target); + + /* Log minimal metadata now, so it is not lost if the system is about to shut down. */ + log_info("Process "PID_FMT" (%s) of user "UID_FMT" terminated abnormally with signal %i/%s, processing...", + context.pidref.pid, context.comm, context.uid, context.signo, + signal_to_string(context.signo)); + + r = coredump_handshake(&context); + if (r < 0) + return r; + + (void) fd_nonblock(context.input_fd, false); + + if (coredump_send_to_container(&context) > 0) + return 0; + + return coredump_submit(config, &context); +} diff --git a/src/coredump/coredump-worker.h b/src/coredump/coredump-worker.h new file mode 100644 index 0000000000..2d581b2986 --- /dev/null +++ b/src/coredump/coredump-worker.h @@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +#pragma once + +#include "coredump-forward.h" + +int coredump_worker(const CoredumpConfig *config, int coredump_fd, usec_t timestamp); diff --git a/src/coredump/coredump.c b/src/coredump/coredump.c index 649756d332..bbb59f7695 100644 --- a/src/coredump/coredump.c +++ b/src/coredump/coredump.c @@ -4,6 +4,7 @@ #include "coredump-backtrace.h" #include "coredump-kernel-helper.h" +#include "coredump-manager.h" #include "coredump-receive.h" #include "coredump-util.h" #include "dlopen-note.h" @@ -25,14 +26,22 @@ static int run(int argc, char *argv[]) { if (streq_ptr(argv[1], "--backtrace")) return coredump_backtrace(argc, argv); + if (streq_ptr(argv[1], "--check-requirements")) { + log_setup(); + return coredump_manager_check_requirements(LOG_DEBUG); + } + + /* Make sure we never enter a loop. */ + (void) set_dumpable(SUID_DUMP_DISABLE); + + if (streq_ptr(argv[1], "--manager")) + return coredump_manager(); + /* First, log to a safe place, since we don't know what crashed and it might be journald which we'd * rather not log to then. */ log_parse_environment(); log_set_target_and_open(LOG_TARGET_KMSG); - /* Make sure we never enter a loop. */ - (void) set_dumpable(SUID_DUMP_DISABLE); - r = sd_listen_fds(false); if (r < 0) return log_error_errno(r, "Failed to determine the number of file descriptors: %m"); diff --git a/src/coredump/meson.build b/src/coredump/meson.build index 084088dde8..e36099c51b 100644 --- a/src/coredump/meson.build +++ b/src/coredump/meson.build @@ -10,9 +10,11 @@ systemd_coredump_sources = files( 'coredump-config.c', 'coredump-context.c', 'coredump-kernel-helper.c', + 'coredump-manager.c', 'coredump-receive.c', 'coredump-send.c', 'coredump-submit.c', + 'coredump-worker.c', ) systemd_coredump_export_sources = files( 'coredump-vacuum.c', diff --git a/src/include/override/sched.h b/src/include/override/sched.h index 3eed393118..03ded107a5 100644 --- a/src/include/override/sched.h +++ b/src/include/override/sched.h @@ -19,6 +19,13 @@ static_assert(CLONE_NEWTIME == 0x00000080, ""); #endif +/* Not exposed yet. Defined at include/linux/sched.h */ +#ifndef PF_DUMPCORE +# define PF_DUMPCORE 0x00000200 +#else +static_assert(PF_DUMPCORE == 0x00000200, ""); +#endif + /* Not exposed yet. Defined at include/linux/sched.h */ #ifndef PF_KTHREAD # define PF_KTHREAD 0x00200000 diff --git a/src/journal/journald-native.c b/src/journal/journald-native.c index 01e7d65453..66a20a7514 100644 --- a/src/journal/journald-native.c +++ b/src/journal/journald-native.c @@ -377,7 +377,9 @@ static size_t entry_size_max_by_ucred(Manager *m, const struct ucred *ucred, con log_ratelimit_warning_errno(r, JOURNAL_LOG_RATELIMIT, "Failed to retrieve credentials for PID " PID_FMT ", ignoring: %m", ucred->pid); - else if (context->unit && startswith(context->unit, "systemd-coredump@")) + else if (context->unit && + (startswith(context->unit, "systemd-coredump@") || + streq(context->unit, "systemd-coredumpd.service"))) return ENTRY_SIZE_MAX; } diff --git a/test/units/TEST-87-AUX-UTILS-VM.coredump.sh b/test/units/TEST-87-AUX-UTILS-VM.coredump.sh index ae7b38ab02..b24d04a7ce 100755 --- a/test/units/TEST-87-AUX-UTILS-VM.coredump.sh +++ b/test/units/TEST-87-AUX-UTILS-VM.coredump.sh @@ -17,6 +17,12 @@ at_exit() { rm -fv -- "$CORE_TEST_BIN" "$CORE_TEST_UNPRIV_BIN" "$MAKE_DUMP_SCRIPT" } +maybe_reload_coredumpd() { + if systemctl -q is-active systemd-coredumpd.service; then + systemctl reload systemd-coredumpd.service + fi +} + (! systemd-detect-virt -cq) trap at_exit EXIT @@ -25,8 +31,31 @@ trap at_exit EXIT journalctl --sync journalctl --rotate +if /usr/lib/systemd/systemd-coredump --check-requirements && systemctl is-enabled systemd-coredumpd.service; then + systemctl is-active systemd-coredumpd.service +fi + # Check that we're the ones to receive coredumps -sysctl kernel.core_pattern | grep systemd-coredump +if systemctl -q is-active systemd-coredumpd.service; then + systemctl is-active systemd-coredump-register.service + [[ -e /run/sysctl.d/60-coredump.conf ]] + sysctl kernel.core_pattern | grep '@@/run/systemd/coredumpd/kernel' + + systemctl stop systemd-coredumpd.service + (! systemctl is-active systemd-coredump-register.service) + [[ ! -e /run/sysctl.d/60-coredump.conf ]] + sysctl kernel.core_pattern | grep 'systemd-coredump' + + systemctl start systemd-coredumpd.service + timeout 30 bash -xec 'until systemctl -q is-active systemd-coredump-register.service; do sleep 1; done' + [[ -e /run/sysctl.d/60-coredump.conf ]] + sysctl kernel.core_pattern | grep '@@/run/systemd/coredumpd/kernel' +else + assert_eq "$(systemctl is-active systemd-coredumpd.service ||:)" "inactive" + assert_eq "$(systemctl is-active systemd-coredump-register.service ||:)" "inactive" + [[ ! -e /run/sysctl.d/60-coredump.conf ]] + sysctl kernel.core_pattern | grep 'systemd-coredump' +fi # Prepare "fake" binaries for coredumps, so we can properly exercise # the matching stuff too @@ -70,11 +99,13 @@ chmod +x "$MAKE_DUMP_SCRIPT" # with Storage=external as well mkdir -p /run/systemd/coredump.conf.d/ printf '[Coredump]\nStorage=external' >/run/systemd/coredump.conf.d/99-external.conf +maybe_reload_coredumpd "$MAKE_DUMP_SCRIPT" "$CORE_TEST_BIN" "SIGTRAP" "$MAKE_DUMP_SCRIPT" "$CORE_TEST_BIN" "SIGABRT" -rm -fv /run/systemd/coredump.conf.d/99-external.conf # Wait a bit for the coredumps to get processed timeout 30 bash -c "while [[ \$(coredumpctl list -q --no-legend $CORE_TEST_BIN | wc -l) -lt 4 ]]; do sleep 1; done" +rm -fv /run/systemd/coredump.conf.d/99-external.conf +maybe_reload_coredumpd if cgroupfs_supports_user_xattrs; then # Make sure we can forward crashes back to containers @@ -192,11 +223,13 @@ UNPRIV_CMD=(systemd-run --user --wait --pipe -M "testuser@.host" -E SYSTEMD_PAGE # with Storage=external as well mkdir -p /run/systemd/coredump.conf.d/ printf '[Coredump]\nStorage=external' >/run/systemd/coredump.conf.d/99-external.conf +maybe_reload_coredumpd "${UNPRIV_CMD[@]}" "$MAKE_DUMP_SCRIPT" "$CORE_TEST_UNPRIV_BIN" "SIGTRAP" "${UNPRIV_CMD[@]}" "$MAKE_DUMP_SCRIPT" "$CORE_TEST_UNPRIV_BIN" "SIGABRT" -rm -fv /run/systemd/coredump.conf.d/99-external.conf # Wait a bit for the coredumps to get processed timeout 30 bash -c "while [[ \$(coredumpctl list -q --no-legend $CORE_TEST_UNPRIV_BIN | wc -l) -lt 4 ]]; do sleep 1; done" +rm -fv /run/systemd/coredump.conf.d/99-external.conf +maybe_reload_coredumpd # Sync and rotate journal again to make the coredump stored in an archived journal. journalctl --sync @@ -317,6 +350,7 @@ EOF mkdir -p /run/systemd/coredump.conf.d/ printf '[Coredump]\nEnterNamespace=no' >/run/systemd/coredump.conf.d/99-enter-namespace.conf + maybe_reload_coredumpd unshare --pid --fork --mount-proc --mount --uts --ipc --net "$MAKE_STACKTRACE_DUMP" "test-stacktrace-not-symbolized" timeout 30 bash -c "until coredumpctl list -q --no-legend /tmp/test-stacktrace-not-symbolized; do sleep .2; done" @@ -327,6 +361,7 @@ EOF (! grep -E "#[0-9]+ .* baz " /tmp/not-symbolized.log) printf '[Coredump]\nEnterNamespace=yes' >/run/systemd/coredump.conf.d/99-enter-namespace.conf + maybe_reload_coredumpd unshare --pid --fork --mount-proc --mount --uts --ipc --net "$MAKE_STACKTRACE_DUMP" "test-stacktrace-symbolized" timeout 30 bash -c "until coredumpctl list -q --no-legend /tmp/test-stacktrace-symbolized; do sleep .2; done" coredumpctl info /tmp/test-stacktrace-symbolized | tee /tmp/symbolized.log @@ -337,6 +372,7 @@ EOF test -d /usr/lib/debug/ && umount /usr/lib/debug/ rm -f "$MAKE_STACKTRACE_DUMP" /run/systemd/coredump.conf.d/99-enter-namespace.conf /tmp/{not-,}symbolized.log + maybe_reload_coredumpd else echo "libdw doesn't not support setting sysroot, skipping EnterNamespace= test" fi diff --git a/tools/command_ignorelist b/tools/command_ignorelist index cb256d6647..69aaccb96c 100644 --- a/tools/command_ignorelist +++ b/tools/command_ignorelist @@ -157,6 +157,9 @@ systemd-cgtop.xml ./refsect1[title="Keys"]/variablelist/varlistentry[term="h"] systemd-cgtop.xml ./refsect1[title="Keys"]/variablelist/varlistentry[term="q"] systemd-cgtop.xml ./refsect1[title="Keys"]/variablelist/varlistentry[term="p"] systemd-cgtop.xml ./refsect1[title="Keys"]/variablelist/varlistentry[term="+"] +systemd-coredump.xml ./refsect1[title="Description"]/refsect2[title="Command-line Options"]/variablelist/varlistentry[term="(no option)"] +systemd-coredump.xml ./refsect1[title="Description"]/refsect2[title="Command-line Options"]/variablelist/varlistentry[term="--backtrace"] +systemd-coredump.xml ./refsect1[title="Description"]/refsect2[title="Units"]/variablelist/varlistentry[term="systemd-coredump.socket"] systemd-delta.xml ./refsect1[title="Options"]/variablelist/varlistentry[term="-t"] systemd-delta.xml ./refsect1[title="Options"]/variablelist/varlistentry[term="--diff="] systemd-delta.xml ./refsect1[title="Options"]/variablelist/varlistentry[term="-t"]/listitem/para/variablelist/varlistentry[term="masked"] diff --git a/units/meson.build b/units/meson.build index 6e0c474830..7e2dee392f 100644 --- a/units/meson.build +++ b/units/meson.build @@ -342,6 +342,15 @@ units = [ 'file' : 'systemd-coredump@.service.in', 'conditions' : ['ENABLE_COREDUMP'], }, + { + 'file' : 'systemd-coredump-register.service.in', + 'conditions' : ['ENABLE_COREDUMP'], + }, + { + 'file' : 'systemd-coredumpd.service.in', + 'conditions' : ['ENABLE_COREDUMP'], + 'symlinks' : ['sysinit.target.wants/'], + }, { 'file' : 'systemd-creds.socket', 'symlinks' : ['sockets.target.wants/'], diff --git a/units/systemd-coredump-register.service.in b/units/systemd-coredump-register.service.in new file mode 100644 index 0000000000..2e3d3651d1 --- /dev/null +++ b/units/systemd-coredump-register.service.in @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +[Unit] +Description=Kernel Core Pattern Register +Documentation=man:systemd-coredump(8) + +DefaultDependencies=no +After=systemd-coredumpd.service systemd-sysctl.service +BindsTo=systemd-coredumpd.service +Before=sysinit.target shutdown.target +Conflicts=shutdown.target + +ConditionVirtualization=!container +RefuseManualStart=yes +RefuseManualStop=yes + +[Service] +Type=oneshot +ExecCondition={{LIBEXECDIR}}/systemd-coredump --check-requirements +ExecStart={{LIBEXECDIR}}/systemd-sysctl --strict --verify --save=60-coredump --inline "kernel.core_pattern=@@/run/systemd/coredumpd/kernel" +ExecStop={{LIBEXECDIR}}/systemd-sysctl --revert=60-coredump --prefix=/kernel/core_pattern +RemainAfterExit=yes diff --git a/units/systemd-coredumpd.service.in b/units/systemd-coredumpd.service.in new file mode 100644 index 0000000000..01eecb173a --- /dev/null +++ b/units/systemd-coredumpd.service.in @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: LGPL-2.1-or-later +# +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +[Unit] +Description=Listen Kernel Core Dump Socket +Documentation=man:systemd-coredump(8) + +DefaultDependencies=no +After=local-fs.target systemd-journald.socket +Before=shutdown.target sysinit.target systemd-coredump-register.service +Conflicts=shutdown.target +Requires=systemd-journald.socket +Upholds=systemd-coredump-register.service + +ConditionVirtualization=!container + +[Service] +Type=notify-reload +ExecCondition={{LIBEXECDIR}}/systemd-coredump --check-requirements +ExecStart={{LIBEXECDIR}}/systemd-coredump --manager + +CapabilityBoundingSet=CAP_CHOWN CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_FOWNER CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_SYS_PTRACE +FileDescriptorStoreMax=512 +FileDescriptorStorePreserve=yes +IPAddressDeny=any +KillMode=mixed +LockPersonality=yes +MemoryDenyWriteExecute=yes +Nice=9 +NoNewPrivileges=yes +OOMScoreAdjust=-1000 +PrivateDevices=yes +PrivateNetwork=yes +PrivateTmp=disconnected +ProtectClock=yes +ProtectControlGroups=yes +ProtectHome=read-only +ProtectHostname=yes +ProtectKernelModules=yes +ProtectKernelTunables=yes +ProtectSystem=strict +Restart=on-failure +RestartSec=0 +RestrictAddressFamilies=AF_UNIX +RestrictRealtime=yes +RuntimeDirectory=systemd/coredumpd +RuntimeDirectoryPreserve=yes +StateDirectory=systemd/coredump +SystemCallArchitectures=native +SystemCallErrorNumber=EPERM +SystemCallFilter=@system-service @mount +TimeoutStopSec=6min +{{SERVICE_WATCHDOG}}