During daemon-reload (or daemon-reexec), when a unit becomes an alias to
another unit, deserialising the alias's stale serialised state can
corrupt the canonical unit's live runtime state.
Consider this scenario:
1. Before reload:
- a.service is running
- b.service was stopped earlier and is dead
- Both exist as independent units
2. User creates an alias to migrate from b -> a:
- `ln -s /run/systemd/system/a.service /etc/systemd/system/b.service`
3. daemon-reload triggers serialisation. State file contains both units:
- a.service -> state=running, cgroup=/system.slice/a.service,
PID=1234, ...
- b.service -> state=dead, cgroup=(empty), no PIDs, ...
4. During deserialisation:
- Processes a.service: loads Unit A, deserialises -> state=RUNNING
- Processes b.service: manager_load_unit() detects symlink, returns
Unit A
- unit_deserialize_state(Unit A, ...) overwrites with b's dead state
5. The result is that:
- Unit A incorrectly shows state=dead despite PID 1234 still running
- If a.service has Upholds= dependents, catch-up logic sees
a.service should be running but is dead
- systemd starts a.service again -> PID 5678
- Two instances run: PID 1234 (left-over) and PID 5678 (new)
This bug is deterministic when serialisation orders a.service before
b.service.
The root cause is that manager_deserialize_one_unit() calls
manager_load_unit(name, &u) which resolves aliases via
unit_follow_merge(), returning the canonical Unit object. However, the
code doesn't distinguish between two cases when u->id differs from the
requested name from the state file. In the corruption case, we're
deserialising an alias entry and unit_deserialize_state() blindly
overwrites the canonical unit's fields with stale data from the old,
independent unit. The serialised b.service then overwrites Unit A's
correct live state.
This commit first scans the serialised unit names, then adds a check
after manager_load_unit():
if (!streq(u->id, name) && set_contains(serialized_units, u->id))
...
This detects when the loaded unit's canonical ID (u->id) differs from
the serialised name, indicating the name is now an alias for a different
unit and the canonical unit also has its own serialised state entry.
If the canonical unit does not have its own serialised state entry, we
keep the state entry. That handles cases where the old name is really
just a rename, and thus the old name is the only serialised state for
the unit. In that case there is no bug, because there is no separate
canonical state entry for the stale alias entry to overwrite.
Skipping is safe because:
1. The canonical unit's own state entry will be correctly deserialised
regardless of order. This fix only prevents other stale alias entries
from corrupting it.
2. unit_merge() has already transferred the necessary data. When
b.service became an alias during unit loading, unit_merge() already
migrated dependencies and references to the canonical unit.
3. After merging, the alias doesn't have its own runtime state. The
serialised data represents b.service when it was independent, which
is now obsolete once the canonical unit also has its own serialised
entry.
4. All fields are stale. unit_deserialize_state() would overwrite state,
timestamps, cgroup paths, pids, etc. There's no scenario where we
want this data applied on top of the canonical unit's own serialised
state.
This fix also correctly handles unit precedence. For example, imagine
this scenario:
1. `b.service` is a valid, running unit defined in `/run`.
2. The sysadmin creates `ln -s .../a.service /etc/.../b.service`.
3. On reload, the new symlink in `/etc` overrides the unit in `/run`.
The new perspective from the manager side is that `b.service` is now an
alias for `a.service`.
In this case, systemd correctly abandons the old b.service unit, because
that's the intended general semantics of unit file precedence. We also
do that in other cases, like when a unit file in /etc/systemd/system/
masks a vendor-supplied unit file in /lib/systemd/system/, or when an
admin uses systemctl mask to explicitly disable a unit.
In all these scenarios, the configuration with the highest precedence
(in /etc/) is treated as the new source of truth. The old unit's
definition is discarded, and its running processes are (correctly)
abandoned. In that respect we are not doing anything new here.
Some may ask why we shouldn't just ignore the symlink if we think this
case will come up. I think there are multiple very strong reasons not to
do so:
1. It violates unit precedence. The unit design is built on a strict
precedence list. When an admin puts any file in /etc, they are
intentionally overriding everything else. If manager_load_unit were
to "ignore" this file based on runtime state, it would break this
fundamental precedent.
2. It makes daemon-reload stateful. daemon-reload is supposed to be a
simple, stateless operation, basically to read the files on disk and
apply the new configuration. But doing this would make daemon-reload
stateful, because we'd have to read the files on disk, but
cross-reference the current runtime state, and... maybe ignore some
files. This is complex and unpredictable.
3. It also completely ignores the user intent. The admin clearly has
tried to replace the old service with an alias. Ignoring their
instruction is the opposite of what they want.
Fixes: https://github.com/systemd/systemd/issues/38817
Fixes: https://github.com/systemd/systemd/issues/37482
230 lines
7.2 KiB
Bash
Executable File
230 lines
7.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# SPDX-License-Identifier: LGPL-2.1-or-later
|
|
set -eux
|
|
set -o pipefail
|
|
|
|
# Verify that stale alias state doesn't overwrite canonical unit state.
|
|
# 1. Legit unit is running (PID A).
|
|
# 2. Sus units are running (PID B, C, D...).
|
|
# 3. We alias sus -> legit.
|
|
# 4. If the bug triggers, legit unit's state is overwritten by a sus unit's state.
|
|
# 5. Legit unit thinks it is now PID B (or C, or D...).
|
|
# 6. We detect this PID change as proof of corruption.
|
|
|
|
declare -a abandoned_pids=()
|
|
|
|
reap_abandoned_pids() {
|
|
local pid attempt
|
|
|
|
if (( ${#abandoned_pids[@]} == 0 )); then
|
|
return 0
|
|
fi
|
|
|
|
echo "Reaping ${#abandoned_pids[@]} abandoned processes..."
|
|
|
|
for pid in "${abandoned_pids[@]}"; do
|
|
kill "$pid" 2>/dev/null || true
|
|
done
|
|
|
|
for pid in "${abandoned_pids[@]}"; do
|
|
for attempt in $(seq 1 50); do
|
|
if ! kill -0 "$pid" 2>/dev/null; then
|
|
break
|
|
fi
|
|
|
|
sleep 0.1
|
|
done
|
|
|
|
if kill -0 "$pid" 2>/dev/null; then
|
|
kill -KILL "$pid" 2>/dev/null || true
|
|
fi
|
|
|
|
for attempt in $(seq 1 50); do
|
|
if ! kill -0 "$pid" 2>/dev/null; then
|
|
break
|
|
fi
|
|
|
|
sleep 0.1
|
|
done
|
|
|
|
if kill -0 "$pid" 2>/dev/null; then
|
|
echo "ERROR: Failed to reap abandoned process PID $pid"
|
|
return 1
|
|
fi
|
|
done
|
|
|
|
abandoned_pids=()
|
|
}
|
|
|
|
run_test() {
|
|
local reload_cmd="${1:?}"
|
|
local current_pid journal_warnings new_pid orig_pid pid reload_start unit warning_count
|
|
|
|
echo ""
|
|
echo "========================================="
|
|
echo "Testing with: systemctl $reload_cmd"
|
|
echo "========================================="
|
|
|
|
cat >/run/systemd/system/legit.service <<'EOF'
|
|
[Service]
|
|
Type=simple
|
|
ExecStart=/bin/sleep infinity
|
|
EOF
|
|
|
|
# Create 20 sus units. They must be Type=simple/running so systemd
|
|
# CANNOT garbage collect them. If they are dead/stopped, systemd can remove
|
|
# them from memory before serialization
|
|
echo "Creating 20 sus units..."
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
cat >/run/systemd/system/sus-"${i}".service <<'EOF'
|
|
[Service]
|
|
Type=simple
|
|
ExecStart=/bin/sleep infinity
|
|
EOF
|
|
done
|
|
|
|
systemctl daemon-reload
|
|
|
|
echo "Starting legit unit..."
|
|
systemctl start legit.service
|
|
|
|
echo "Starting sus units..."
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
systemctl start sus-"${i}".service
|
|
done
|
|
|
|
echo "Setup complete: 1 running legit unit, 20 running sus units"
|
|
|
|
orig_pid=$(systemctl show -P MainPID legit.service)
|
|
echo "Original legit PID: $orig_pid"
|
|
|
|
if (( orig_pid == 0 )); then
|
|
echo "Error: Legit PID is 0, setup failed."
|
|
return 1
|
|
fi
|
|
|
|
# Since ordering is not deterministic we should loop 3 times to reduce
|
|
# false negative rate (ordering luck). With this it's roughly 0.01% chance
|
|
# of falsely passing. Falsely failing does not happen, though.
|
|
for attempt in 1 2 3; do
|
|
echo ""
|
|
echo "--- Attempt $attempt/3 ---"
|
|
|
|
unset sus_pids
|
|
declare -A sus_pids
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
pid=$(systemctl show -P MainPID sus-"${i}".service)
|
|
if (( pid != 0 )); then
|
|
sus_pids["sus-${i}"]=$pid
|
|
abandoned_pids+=("$pid")
|
|
echo "sus-${i}.service PID: $pid"
|
|
fi
|
|
done
|
|
|
|
echo "Converting sus units to symlinks -> legit.service..."
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
rm -f /run/systemd/system/sus-"${i}".service
|
|
ln -sf /run/systemd/system/legit.service /run/systemd/system/sus-"${i}".service
|
|
done
|
|
|
|
reload_start=$(date '+%Y-%m-%d %H:%M:%S')
|
|
|
|
echo "Running $reload_cmd..."
|
|
systemctl "$reload_cmd"
|
|
|
|
# If the bug triggered, legit.service deserialized a sus unit's state
|
|
# and overwrote its own MainPID with the sus unit's PID.
|
|
new_pid=$(systemctl show -P MainPID legit.service)
|
|
|
|
if [[ "$new_pid" != "$orig_pid" ]]; then
|
|
echo "legit.service PID changed from $orig_pid to $new_pid!"
|
|
echo "The stale alias state corrupted the canonical unit."
|
|
return 1
|
|
fi
|
|
|
|
echo "legit.service PID remains $new_pid. Attempt $attempt passed."
|
|
|
|
# Verify that all sus unit processes were abandoned (still running but no longer tracked)
|
|
echo "Verifying sus unit processes were abandoned..."
|
|
for unit in "${!sus_pids[@]}"; do
|
|
pid=${sus_pids[$unit]}
|
|
# Process should still be running
|
|
if ! kill -0 "$pid" 2>/dev/null; then
|
|
echo "ERROR: $unit process (PID $pid) was killed instead of abandoned!"
|
|
return 1
|
|
fi
|
|
# But the alias should now either be inactive (MainPID=0) or resolve to legit's PID.
|
|
current_pid=$(systemctl show -P MainPID "${unit}.service")
|
|
if ! (( current_pid == 0 || current_pid == new_pid )); then
|
|
echo "ERROR: $unit unexpectedly reports MainPID=$current_pid after aliasing!"
|
|
return 1
|
|
fi
|
|
echo "$unit process (PID $pid) was correctly abandoned (still running, no longer tracked)"
|
|
done
|
|
|
|
# Check consistency between journal warnings and abandoned processes
|
|
echo "Checking journal for stale state warnings..."
|
|
journal_warnings=$(journalctl --since "$reload_start" --no-pager | grep "Skipping stale state" || true)
|
|
warning_count=$(echo "$journal_warnings" | grep -c "Skipping stale state" || true)
|
|
|
|
echo "Found $warning_count 'Skipping stale state' warnings"
|
|
|
|
# Extract unit names from warnings and verify they match our sus units
|
|
if (( warning_count > 0 )); then
|
|
echo "Verifying warning consistency..."
|
|
for unit in "${!sus_pids[@]}"; do
|
|
if [[ "$journal_warnings" != *"${unit}.service"* ]]; then
|
|
echo "WARNING: Expected journal warning for ${unit}.service but didn't find it"
|
|
fi
|
|
done
|
|
fi
|
|
|
|
reap_abandoned_pids
|
|
|
|
if (( attempt < 3 )); then
|
|
echo "Resetting sus units..."
|
|
|
|
# We must fully reset to get independent running units again
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
rm -f /run/systemd/system/sus-"${i}".service
|
|
cat >/run/systemd/system/sus-"${i}".service <<'EOF'
|
|
[Service]
|
|
Type=simple
|
|
ExecStart=/bin/sleep infinity
|
|
EOF
|
|
done
|
|
|
|
systemctl "$reload_cmd"
|
|
|
|
# Ensure they are running again (they might have been
|
|
# abandoned/killed during the transition)
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
systemctl start sus-"${i}".service
|
|
done
|
|
|
|
echo "Reset complete."
|
|
fi
|
|
done
|
|
|
|
echo "legit.service did not become sus through all 3 $reload_cmd cycles"
|
|
|
|
echo "$reload_cmd test passed"
|
|
}
|
|
|
|
cleanup_test_units() {
|
|
reap_abandoned_pids || true
|
|
systemctl stop legit.service 2>/dev/null || true
|
|
for i in $(seq -f "%02g" 1 20); do
|
|
systemctl stop sus-"${i}".service 2>/dev/null || true
|
|
rm -f /run/systemd/system/sus-"${i}".service
|
|
done
|
|
rm -f /run/systemd/system/legit.service
|
|
systemctl daemon-reload
|
|
}
|
|
|
|
trap cleanup_test_units EXIT
|
|
|
|
run_test daemon-reload
|
|
cleanup_test_units
|
|
run_test daemon-reexec
|