service: add 'debug' option to RestartMode=

One of the major pait points of managing fleets of headless nodes is
that when something fails at startup, unless debug level was already
enabled (which usually isn't, as it's a firehose), one needs to manually
enable it and pray the issue can be reproduced, which often is really
hard and time consuming, just to get extra info. Usually the extra log
messages are enough to triage an issue.

This new option makes it so that when a service fails and is restarted
due to Restart=, log level for that unit is set to debug, so that all
setup code in pid1 and sd-executor logs at debug level, and also a new
DEBUG_INVOCATION=1 env var is passed to the service itself, so that it
knows it should start with a higher log level. Once the unit succeeds
or reaches the rate limit the original level is restored.
This commit is contained in:
Luca Boccassi
2024-08-27 12:24:45 +01:00
committed by Luca Boccassi
parent 23e3bddc48
commit 7d8bbfbe08
13 changed files with 163 additions and 23 deletions
+21
View File
@@ -165,4 +165,25 @@ rm /run/notify-reload-test.sh
systemd-analyze log-level info
# Ensure that, with system log level info, we get debug level messages when a unit fails to start and is
# restarted with RestartMode=debug
cat >/run/systemd/system/testservice-fail-restart-debug-59.service <<EOF
[Unit]
Description=TEST-59-RELOADING-RESTART Restart=on-failure RestartMode=debug
[Service]
ExecStart=echo hello
Restart=on-failure
RestartMode=debug
StartLimitBurst=3
MountAPIVFS=yes
BindPaths=/nonexistent-debug-59
EOF
systemctl daemon-reload
systemctl start testservice-fail-restart-debug-59.service
wait_on_state_or_fail "testservice-fail-restart-debug-59.service" "failed" "15"
journalctl --sync
journalctl -b | grep -q "Failed to follow symlinks on /nonexistent-debug-59: No such file or directory"
touch /testok