Merge pull request #10244 from poettering/nofile-bump

bump RLIMIT_NOFILE
This commit is contained in:
Lennart Poettering
2018-10-17 17:59:36 +02:00
committed by GitHub
22 changed files with 246 additions and 44 deletions
+1
View File
@@ -10,6 +10,7 @@ Release=27
[Output]
Format=raw_btrfs
Bootable=yes
KernelCommandLine=printk.devkmsg=on
[Partitions]
RootSize=3G
+36
View File
@@ -27,6 +27,42 @@ CHANGES WITH 240 in spe:
non-transient services (i.e. those defined with unit files on disk)
we will continue to default to Type=simple.
* The Linux kernel's current default RLIMIT_NOFILE resource limit for
userspace processes is set to 1024 (soft) and 4096
(hard). Previously, systemd passed this on unmodified to all
processes it forked off. With this systemd release the hard limit
systemd passes on is increased to 256K, overriding the kernel's
defaults and substantially increasing the number of simultaneous file
descriptors unprivileged userspace processes can allocate. Note that
the soft limit remains at 1024 for compatibility reasons: the
traditional UNIX select() call cannot deal with file descriptors >=
1024 and increasing the soft limit globally might thus result in
programs unexpectedly allocating a high file descriptor and thus
failing abnormally when attempting to use it with select() (of
course, programs shouldn't use select() anymore, and prefer
poll()/epoll, but the call unfortunately remains undeservedly popular
at this time). This change reflects the fact that file descriptor
handling in the Linux kernel has been optimized in more recent
kernels and allocating large numbers of them should be much cheaper
both in memory and in performance than it used to be. Programs that
want to take benefit of the increased limit have to "opt-in" into
high file descriptors explicitly by setting their soft limit to the
hard limit during initialization. Of course, when doing that they
must do this acknowledging the fact that they cannot use select()
anymore (and neither can any shared library they use — or any shared
library used by any shared library they use and so on).
* The fs.nr_open and fs.file-max sysctls are now automatically bumped
to the highest possible values, as separate accounting of file
descriptors is no longer necessary, as memcg tracks them correctly as
part of the memory accounting anyway. Thus, from the four limits on
file descriptors currently enforced (fs.file-max, fs.nr_open,
RLIMIT_NOFILE hard, RLIMIT_NOFILE soft) we turn off the first two,
and keep only the latter two. A set of build-time options
(-Dbump-proc-sys-fs-file-max=no and -Dbump-proc-sys-fs-nr-open=no)
has been added to revert this change in behaviour, which might be
an option for systems that turn off memcg in the kernel.
CHANGES WITH 239:
* NETWORK INTERFACE DEVICE NAMING CHANGES: systemd-udevd's "net_id"
+2
View File
@@ -48,6 +48,8 @@ Features:
* optionally: turn on cgroup delegation for per-session scope units
* introduce per-unit (i.e. per-slice, per-service) journal log size limits.
* optionally, if a per-partition GPT flag is set for the root/home/… partitions
format the partition on next boot and unset the flag, in order to implement
factory reset. also, add a second flag that simply indicates whether such a
+6 -1
View File
@@ -73,6 +73,10 @@ sysvrcnd_path = get_option('sysvrcnd-path')
conf.set10('HAVE_SYSV_COMPAT', sysvinit_path != '' and sysvrcnd_path != '',
description : 'SysV init scripts and rcN.d links are supported')
conf.set10('BUMP_PROC_SYS_FS_FILE_MAX', get_option('bump-proc-sys-fs-file-max'))
conf.set10('BUMP_PROC_SYS_FS_NR_OPEN', get_option('bump-proc-sys-fs-nr-open'))
conf.set('HIGH_RLIMIT_NOFILE', 256*1024)
# join_paths ignore the preceding arguments if an absolute component is
# encountered, so this should canonicalize various paths when they are
# absolute or relative.
@@ -227,7 +231,7 @@ conf.set_quoted('SYSTEMD_EXPORT_PATH', join_paths(rootlib
conf.set_quoted('VENDOR_KEYRING_PATH', join_paths(rootlibexecdir, 'import-pubring.gpg'))
conf.set_quoted('USER_KEYRING_PATH', join_paths(pkgsysconfdir, 'import-pubring.gpg'))
conf.set_quoted('DOCUMENT_ROOT', join_paths(pkgdatadir, 'gatewayd'))
conf.set('MEMORY_ACCOUNTING_DEFAULT', memory_accounting_default ? 'true' : 'false')
conf.set10('MEMORY_ACCOUNTING_DEFAULT', memory_accounting_default)
conf.set_quoted('MEMORY_ACCOUNTING_DEFAULT_YES_NO', memory_accounting_default ? 'yes' : 'no')
substs.set('prefix', prefixdir)
@@ -269,6 +273,7 @@ substs.set('SYSTEM_SYSVRCND_PATH', sysvrcnd_path)
substs.set('RC_LOCAL_SCRIPT_PATH_START', get_option('rc-local'))
substs.set('RC_LOCAL_SCRIPT_PATH_STOP', get_option('halt-local'))
substs.set('MEMORY_ACCOUNTING_DEFAULT', memory_accounting_default ? 'yes' : 'no')
substs.set('HIGH_RLIMIT_NOFILE', conf.get('HIGH_RLIMIT_NOFILE'))
#####################################################################
+4
View File
@@ -49,6 +49,10 @@ option('debug-extra', type : 'array', choices : ['hashmap', 'mmap-cache'], value
description : 'enable extra debugging')
option('memory-accounting-default', type : 'boolean',
description : 'enable MemoryAccounting= by default')
option('bump-proc-sys-fs-file-max', type : 'boolean',
description : 'bump /proc/sys/fs/file-max to ULONG_MAX')
option('bump-proc-sys-fs-nr-open', type : 'boolean',
description : 'bump /proc/sys/fs/nr_open to INT_MAX')
option('valgrind', type : 'boolean', value : false,
description : 'do extra operations to avoid valgrind warnings')
option('log-trace', type : 'boolean', value : false,
+2
View File
@@ -75,3 +75,5 @@
_CONF_PATHS_SPLIT_USR(n))
#define LONG_LINE_MAX (1U*1024U*1024U)
#define HIGH_RLIMIT_MEMLOCK (1024ULL*1024ULL*64ULL)
+31 -2
View File
@@ -5,6 +5,7 @@
#include "alloc-util.h"
#include "extract-word.h"
#include "fd-util.h"
#include "format-util.h"
#include "macro.h"
#include "missing.h"
@@ -33,8 +34,15 @@ int setrlimit_closest(int resource, const struct rlimit *rlim) {
if (highest.rlim_max == RLIM_INFINITY)
return -EPERM;
fixed.rlim_cur = MIN(rlim->rlim_cur, highest.rlim_max);
fixed.rlim_max = MIN(rlim->rlim_max, highest.rlim_max);
fixed = (struct rlimit) {
.rlim_cur = MIN(rlim->rlim_cur, highest.rlim_max),
.rlim_max = MIN(rlim->rlim_max, highest.rlim_max),
};
/* Shortcut things if we wouldn't change anything. */
if (fixed.rlim_cur == highest.rlim_cur &&
fixed.rlim_max == highest.rlim_max)
return 0;
if (setrlimit(resource, &fixed) < 0)
return -errno;
@@ -360,3 +368,24 @@ void rlimit_free_all(struct rlimit **rl) {
for (i = 0; i < _RLIMIT_MAX; i++)
rl[i] = mfree(rl[i]);
}
int rlimit_nofile_bump(int limit) {
int r;
/* Bumps the (soft) RLIMIT_NOFILE resource limit as close as possible to the specified limit. If a negative
* limit is specified, bumps it to the maximum the kernel and the hard resource limit allows. This call should
* be used by all our programs that might need a lot of fds, and that know how to deal with high fd numbers
* (i.e. do not use select() — which chokes on fds >= 1024) */
if (limit < 0)
limit = read_nr_open();
if (limit < 3)
limit = 3;
r = setrlimit_closest(RLIMIT_NOFILE, &RLIMIT_MAKE_CONST(limit));
if (r < 0)
return log_debug_errno(r, "Failed to set RLIMIT_NOFILE: %m");
return 0;
}
+2
View File
@@ -20,3 +20,5 @@ int rlimit_format(const struct rlimit *rl, char **ret);
void rlimit_free_all(struct rlimit **rl);
#define RLIMIT_MAKE_CONST(lim) ((struct rlimit) { lim, lim })
int rlimit_nofile_bump(int limit);
+112 -15
View File
@@ -73,6 +73,7 @@
#include "stdio-util.h"
#include "strv.h"
#include "switch-root.h"
#include "sysctl-util.h"
#include "terminal-util.h"
#include "umask-util.h"
#include "user-util.h"
@@ -1162,18 +1163,102 @@ static int prepare_reexecute(
return 0;
}
static void bump_file_max_and_nr_open(void) {
/* Let's bump fs.file-max and fs.nr_open to their respective maximums. On current kernels large numbers of file
* descriptors are no longer a performance problem and their memory is properly tracked by memcg, thus counting
* them and limiting them in another two layers of limits is unnecessary and just complicates things. This
* function hence turns off 2 of the 4 levels of limits on file descriptors, and makes RLIMIT_NOLIMIT (soft +
* hard) the only ones that really matter. */
#if BUMP_PROC_SYS_FS_FILE_MAX || BUMP_PROC_SYS_FS_NR_OPEN
_cleanup_free_ char *t = NULL;
int r;
#endif
#if BUMP_PROC_SYS_FS_FILE_MAX
/* I so wanted to use STRINGIFY(ULONG_MAX) here, but alas we can't as glibc/gcc define that as
* "(0x7fffffffffffffffL * 2UL + 1UL)". Seriously. 😢 */
if (asprintf(&t, "%lu\n", ULONG_MAX) < 0) {
log_oom();
return;
}
r = sysctl_write("fs/file-max", t);
if (r < 0)
log_full_errno(IN_SET(r, -EROFS, -EPERM, -EACCES) ? LOG_DEBUG : LOG_WARNING, r, "Failed to bump fs.file-max, ignoring: %m");
#endif
#if BUMP_PROC_SYS_FS_FILE_MAX && BUMP_PROC_SYS_FS_NR_OPEN
t = mfree(t);
#endif
#if BUMP_PROC_SYS_FS_NR_OPEN
int v = INT_MAX;
/* Arg! The kernel enforces maximum and minimum values on the fs.nr_open, but we don't really know what they
* are. The expression by which the maximum is determined is dependent on the architecture, and is something we
* don't really want to copy to userspace, as it is dependent on implementation details of the kernel. Since
* the kernel doesn't expose the maximum value to us, we can only try and hope. Hence, let's start with
* INT_MAX, and then keep halving the value until we find one that works. Ugly? Yes, absolutely, but kernel
* APIs are kernel APIs, so what do can we do... 🤯 */
for (;;) {
int k;
v &= ~(__SIZEOF_POINTER__ - 1); /* Round down to next multiple of the pointer size */
if (v < 1024) {
log_warning("Can't bump fs.nr_open, value too small.");
break;
}
k = read_nr_open();
if (k < 0) {
log_error_errno(k, "Failed to read fs.nr_open: %m");
break;
}
if (k >= v) { /* Already larger */
log_debug("Skipping bump, value is already larger.");
break;
}
if (asprintf(&t, "%i\n", v) < 0) {
log_oom();
return;
}
r = sysctl_write("fs/nr_open", t);
t = mfree(t);
if (r == -EINVAL) {
log_debug("Couldn't write fs.nr_open as %i, halving it.", v);
v /= 2;
continue;
}
if (r < 0) {
log_full_errno(IN_SET(r, -EROFS, -EPERM, -EACCES) ? LOG_DEBUG : LOG_WARNING, r, "Failed to bump fs.nr_open, ignoring: %m");
break;
}
log_debug("Successfully bumped fs.nr_open to %i", v);
break;
}
#endif
}
static int bump_rlimit_nofile(struct rlimit *saved_rlimit) {
int r, nr;
assert(saved_rlimit);
/* Save the original RLIMIT_NOFILE so that we can reset it
* later when transitioning from the initrd to the main
/* Save the original RLIMIT_NOFILE so that we can reset it later when transitioning from the initrd to the main
* systemd or suchlike. */
if (getrlimit(RLIMIT_NOFILE, saved_rlimit) < 0)
return log_warning_errno(errno, "Reading RLIMIT_NOFILE failed, ignoring: %m");
/* Make sure forked processes get the default kernel setting */
/* Get the underlying absolute limit the kernel enforces */
nr = read_nr_open();
/* Make sure forked processes get limits based on the original kernel setting */
if (!arg_default_rlimit[RLIMIT_NOFILE]) {
struct rlimit *rl;
@@ -1181,11 +1266,25 @@ static int bump_rlimit_nofile(struct rlimit *saved_rlimit) {
if (!rl)
return log_oom();
/* Bump the hard limit for system services to a substantially higher value. The default hard limit
* current kernels set is pretty low (4K), mostly for historical reasons. According to kernel
* developers, the fd handling in recent kernels has been optimized substantially enough, so that we
* can bump the limit now, without paying too high a price in memory or performance. Note however that
* we only bump the hard limit, not the soft limit. That's because select() works the way it works, and
* chokes on fds >= 1024. If we'd bump the soft limit globally, it might accidentally happen to
* unexpecting programs that they get fds higher than what they can process using select(). By only
* bumping the hard limit but leaving the low limit as it is we avoid this pitfall: programs that are
* written by folks aware of the select() problem in mind (and thus use poll()/epoll instead of
* select(), the way everybody should) can explicitly opt into high fds by bumping their soft limit
* beyond 1024, to the hard limit we pass. */
if (arg_system)
rl->rlim_max = MIN((rlim_t) nr, MAX(rl->rlim_max, (rlim_t) HIGH_RLIMIT_NOFILE));
arg_default_rlimit[RLIMIT_NOFILE] = rl;
}
/* Bump up the resource limit for ourselves substantially, all the way to the maximum the kernel allows */
nr = read_nr_open();
/* Bump up the resource limit for ourselves substantially, all the way to the maximum the kernel allows, for
* both hard and soft. */
r = setrlimit_closest(RLIMIT_NOFILE, &RLIMIT_MAKE_CONST(nr));
if (r < 0)
return log_warning_errno(r, "Setting RLIMIT_NOFILE failed, ignoring: %m");
@@ -1197,16 +1296,15 @@ static int bump_rlimit_memlock(struct rlimit *saved_rlimit) {
int r;
assert(saved_rlimit);
assert(getuid() == 0);
/* BPF_MAP_TYPE_LPM_TRIE bpf maps are charged against RLIMIT_MEMLOCK, even though we have CAP_IPC_LOCK which
* should normally disable such checks. We need them to implement IPAccessAllow= and IPAccessDeny=, hence let's
* bump the value high enough for the root user. */
/* BPF_MAP_TYPE_LPM_TRIE bpf maps are charged against RLIMIT_MEMLOCK, even if we have CAP_IPC_LOCK which should
* normally disable such checks. We need them to implement IPAccessAllow= and IPAccessDeny=, hence let's bump
* the value high enough for our user. */
if (getrlimit(RLIMIT_MEMLOCK, saved_rlimit) < 0)
return log_warning_errno(errno, "Reading RLIMIT_MEMLOCK failed, ignoring: %m");
r = setrlimit_closest(RLIMIT_MEMLOCK, &RLIMIT_MAKE_CONST(1024ULL*1024ULL*64ULL));
r = setrlimit_closest(RLIMIT_MEMLOCK, &RLIMIT_MAKE_CONST(HIGH_RLIMIT_MEMLOCK));
if (r < 0)
return log_warning_errno(r, "Setting RLIMIT_MEMLOCK failed, ignoring: %m");
@@ -1868,6 +1966,7 @@ static int initialize_runtime(
machine_id_setup(NULL, arg_machine_id, NULL);
loopback_setup();
bump_unix_max_dgram_qlen();
bump_file_max_and_nr_open();
test_usr();
write_container_id();
}
@@ -1920,11 +2019,9 @@ static int initialize_runtime(
if (prctl(PR_SET_CHILD_SUBREAPER, 1) < 0)
log_warning_errno(errno, "Failed to make us a subreaper: %m");
if (arg_system) {
/* Bump up RLIMIT_NOFILE for systemd itself */
(void) bump_rlimit_nofile(saved_rlimit_nofile);
(void) bump_rlimit_memlock(saved_rlimit_memlock);
}
/* Bump up RLIMIT_NOFILE for systemd itself */
(void) bump_rlimit_nofile(saved_rlimit_nofile);
(void) bump_rlimit_memlock(saved_rlimit_memlock);
return 0;
}
+1 -1
View File
@@ -53,7 +53,7 @@
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitNOFILE=1024:@HIGH_RLIMIT_NOFILE@
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
+5
View File
@@ -15,6 +15,7 @@
#include "bus-error.h"
#include "bus-util.h"
#include "compress.h"
#include "def.h"
#include "fd-util.h"
#include "fileio.h"
#include "fs-util.h"
@@ -26,6 +27,7 @@
#include "parse-util.h"
#include "path-util.h"
#include "process-util.h"
#include "rlimit-util.h"
#include "sigbus.h"
#include "signal-util.h"
#include "string-util.h"
@@ -1067,6 +1069,9 @@ int main(int argc, char *argv[]) {
log_parse_environment();
log_open();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
r = parse_argv(argc, argv);
if (r <= 0)
goto end;
+4
View File
@@ -12,6 +12,7 @@
#include "journal-remote-write.h"
#include "journal-remote.h"
#include "process-util.h"
#include "rlimit-util.h"
#include "signal-util.h"
#include "socket-util.h"
#include "stat-util.h"
@@ -1096,6 +1097,9 @@ int main(int argc, char **argv) {
log_show_color(true);
log_parse_environment();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
r = parse_config();
if (r < 0)
return EXIT_FAILURE;
+4
View File
@@ -20,6 +20,7 @@
#include "mkdir.h"
#include "parse-util.h"
#include "process-util.h"
#include "rlimit-util.h"
#include "sigbus.h"
#include "signal-util.h"
#include "string-util.h"
@@ -780,6 +781,9 @@ int main(int argc, char **argv) {
log_show_color(true);
log_parse_environment();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
r = parse_config();
if (r < 0)
goto finish;
+5 -5
View File
@@ -31,6 +31,7 @@
#include "bus-util.h"
#include "catalog.h"
#include "chattr-util.h"
#include "def.h"
#include "device-private.h"
#include "fd-util.h"
#include "fileio.h"
@@ -2049,6 +2050,10 @@ int main(int argc, char *argv[]) {
log_parse_environment();
log_open();
/* Increase max number of open files if we can, we might needs this when browsing journal files, which might be
* split up into many files. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
r = parse_argv(argc, argv);
if (r <= 0)
goto finish;
@@ -2056,11 +2061,6 @@ int main(int argc, char *argv[]) {
signal(SIGWINCH, columns_lines_cache_reset);
sigbus_install();
/* Increase max number of open files to 16K if we can, we
* might needs this when browsing journal files, which might
* be split up into many files. */
setrlimit_closest(RLIMIT_NOFILE, &RLIMIT_MAKE_CONST(16384));
switch (arg_action) {
case ACTION_NEW_ID128:
+5
View File
@@ -21,6 +21,7 @@
#include "pager.h"
#include "parse-util.h"
#include "process-util.h"
#include "rlimit-util.h"
#include "sigbus.h"
#include "signal-util.h"
#include "spawn-polkit-agent.h"
@@ -1522,6 +1523,10 @@ int main(int argc, char *argv[]) {
setlocale(LC_ALL, "");
log_parse_environment();
log_open();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
sigbus_install();
r = parse_argv(argc, argv);
+6
View File
@@ -22,6 +22,7 @@
#include "cgroup-show.h"
#include "cgroup-util.h"
#include "copy.h"
#include "def.h"
#include "env-util.h"
#include "fd-util.h"
#include "format-table.h"
@@ -37,6 +38,7 @@
#include "path-util.h"
#include "process-util.h"
#include "ptyfwd.h"
#include "rlimit-util.h"
#include "sigbus.h"
#include "signal-util.h"
#include "spawn-polkit-agent.h"
@@ -3030,6 +3032,10 @@ int main(int argc, char*argv[]) {
setlocale(LC_ALL, "");
log_parse_environment();
log_open();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
sigbus_install();
r = parse_argv(argc, argv);
+4 -6
View File
@@ -5264,12 +5264,6 @@ static int show(int argc, char *argv[], void *userdata) {
(void) pager_open(arg_no_pager, false);
if (show_mode == SYSTEMCTL_SHOW_STATUS)
/* Increase max number of open files to 16K if we can, we
* might needs this when browsing journal files, which might
* be split up into many files. */
setrlimit_closest(RLIMIT_NOFILE, &RLIMIT_MAKE_CONST(16384));
/* If no argument is specified inspect the manager itself */
if (show_mode == SYSTEMCTL_SHOW_PROPERTIES && argc <= 1)
return show_one(bus, "/org/freedesktop/systemd1", NULL, show_mode, &new_line, &ellipsized);
@@ -8661,6 +8655,10 @@ int main(int argc, char*argv[]) {
setlocale(LC_ALL, "");
log_parse_environment();
log_open();
/* The journal merging logic potentially needs a lot of fds. */
(void) rlimit_nofile_bump(HIGH_RLIMIT_NOFILE);
sigbus_install();
/* Explicitly not on_tty() to avoid setting cached value.
+3 -3
View File
@@ -30,9 +30,9 @@ RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallArchitectures=native
LockPersonality=yes
# If there are many split upjournal files we need a lot of fds to
# access them all and combine
LimitNOFILE=16384
# If there are many split up journal files we need a lot of fds to access them
# all in parallel.
LimitNOFILE=@HIGH_RLIMIT_NOFILE@
[Install]
Also=systemd-journal-gatewayd.socket
+4
View File
@@ -32,5 +32,9 @@ SystemCallArchitectures=native
LockPersonality=yes
LogsDirectory=journal/remote
# If there are many split up journal files we need a lot of fds to access them
# all in parallel.
LimitNOFILE=@HIGH_RLIMIT_NOFILE@
[Install]
Also=systemd-journal-remote.socket
+3 -3
View File
@@ -32,9 +32,9 @@ SystemCallArchitectures=native
LockPersonality=yes
StateDirectory=systemd/journal-upload
# If there are many split up journal files we need a lot of fds to
# access them all and combine
LimitNOFILE=16384
# If there are many split up journal files we need a lot of fds to access them
# all in parallel.
LimitNOFILE=@HIGH_RLIMIT_NOFILE@
[Install]
WantedBy=multi-user.target
+3 -5
View File
@@ -35,8 +35,6 @@ SystemCallArchitectures=native
LockPersonality=yes
IPAddressDeny=any
# Increase the default a bit in order to allow many simultaneous
# services being run since we keep one fd open per service. Also, when
# flushing journal files to disk, we might need a lot of fds when many
# journal files are combined.
LimitNOFILE=16384
# If there are many split up journal files we need a lot of fds to access them
# all in parallel.
LimitNOFILE=@HIGH_RLIMIT_NOFILE@
+3 -3
View File
@@ -37,6 +37,6 @@ LockPersonality=yes
IPAddressDeny=any
FileDescriptorStoreMax=512
# Increase the default a bit in order to allow many simultaneous
# logins since we keep one fd open per session.
LimitNOFILE=16384
# Increase the default a bit in order to allow many simultaneous logins since
# we keep one fd open per session.
LimitNOFILE=@HIGH_RLIMIT_NOFILE@