Yu Watanabe
79877fdfa9
# Context #40195 defines the initial proposal and the motivation behind this PR. This PR introduces 3 new options for `.link` files `[Link]` section: - `IRQAffinityPolicy=` - `IRQAffinity=` - `IRQAffinityNUMA=` The purpose is to allow `systemd-udevd` to configure a NIC's IRQs affinity to specific CPU(s). `IRQAffinityPolicy=` supports two policies: - `single`: assign all the NIC IRQs to CPU 0, or the first CPU in the CPU set resulting from the union of `IRQAffinity=` and `IRQAffinityNUMA=`. - `spread`: assign all the NIC IRQs to all the CPUs (or the union of `IRQAffinity=` and `IRQAffinityNUMA=` if defined) in a round-robin fashion while optimizing for cache locally while spreading apart queues on CPUs as much as possible. Both `IRQAffinity=` and `IRQAffinityNUMA=` behaves as filters to reduce the CPU set to assign IRQs to, and are only valid if `IRQAffinityPolicy=` is defined. # Spreading IRQs This section describes the algorithm responsible for spreading IRQs over different CPUs to maximize performance. ## 1. Discover CPU topology Read from `/sys/devices/system/cpu/cpu*/topology` to identify: - L3 cache domains (dies) - Physical cores VS hyperthreads - NUMA nodes - Core ordering within each die ``` Example: Dual-socket server with 2 dies per socket, 4 cores per die ┌─────────────────────────────────────────────────────────────────┐ │ NUMA Node 0 │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ Die 0 (L3 Cache) │ │ Die 1 (L3 Cache) │ │ │ │ ┌────┐┌────┐┌────┐┌────│ │ ┌────┐┌────┐┌────┐┌────│ │ │ │ │ C0 ││ C1 ││ C2 ││ C3 │ │ │ C4 ││ C5 ││ C6 ││ C7 │ │ │ │ │0,16││1,17││2,18││3,19│ │ │4,20││5,21││6,22││7,23│ │ │ │ └────┘└────┘└────┘└────│ │ └────┘└────┘└────┘└────│ │ │ └─────────────────────────┘ └─────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ NUMA Node 1 │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ Die 2 (L3 Cache) │ │ Die 3 (L3 Cache) │ │ │ │ ┌────┐┌────┐┌────┐┌────│ │ ┌────┐┌────┐┌────┐┌────│ │ │ │ │ C8 ││ C9 ││C10 ││C11 │ │ │C12 ││C13 ││C14 ││C15 │ │ │ │ │8,24││9,25││10,2││11,2│ │ │12,2││13,2││14,3││15,3│ │ │ │ └────┘└────┘└────┘└────│ │ └────┘└────┘└────┘└────│ │ │ └─────────────────────────┘ └─────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ (Numbers show CPU IDs: first HT, second HT) ``` ## 2. Filter the first hyperthread only Use only the first hyperthread of each physical core to avoid SMT contention. Two IRQs on sibling HTs contend for ALU/cache without cache benefit. ``` Before: CPUs 0-31 (16 cores × 2 HTs) After: CPUs 0-15 (first HT of each core) ``` ## 3. Equidistant permutations Reorder dies and CPUs so consecutive selections are maximally spread apart. ``` Original order: [0, 1, 2, 3] (adjacent dies/CPUs) | v Equidistant: [0, 2, 1, 3] (spread apart) ``` This ensures that even if only 2 IRQs are assigned, they land on dies 0 and 2 (not 0 and 1), maximizing physical distance. The permutation is also applied within each die: ``` Die permutation: Die0 -> Die2 -> Die1 -> Die3 Within each die: ┌───────────────────────────────────────────────┐ │ Die 0: [C0,C1,C2,C3] -> [C0,C2,C1,C3] │ │ Die 1: [C4,C5,C6,C7] -> [C4,C6,C5,C7] │ │ Die 2: [C8,C9,C10,C11] -> [C8,C10,C9,C11] │ │ Die 3: [C12,C13,C14,C15] -> [C12,C14,C13,C15] │ └───────────────────────────────────────────────┘ ``` ## 4. Round-robin selection across dies Pick one CPU from each die in rotation, following permuted order. ``` Round 1: Die0->C0 Die2->C8 Die1->C4 Die3->C12 | | | | v v v v IRQs: [IRQ0] [IRQ1] [IRQ2] [IRQ3] Round 2: Die0->C2 Die2->C10 Die1->C6 Die3->C14 | | | | v v v v IRQs: [IRQ4] [IRQ5] [IRQ6] [IRQ7] ``` If there are more IRQs than physical cores, this logic wraps around and reuse CPUs. Only the first hyperthread of each core is used to avoid cache line contention between queues. Closes #40195.
System and Service Manager
Details
Most documentation is available on systemd's web site.
Assorted, older, general information about systemd can be found in the systemd Wiki.
Information about build requirements is provided in the README file.
Consult our NEWS file for information about what's new in the most recent systemd versions.
Please see the Code Map for information about this repository's layout and content.
Please see the Hacking guide for information on how to hack on systemd and test your modifications.
Please see our Contribution Guidelines for more information about filing GitHub Issues and posting GitHub Pull Requests.
When preparing patches for systemd, please follow our Coding Style Guidelines.
If you are looking for support, please contact our mailing list, join our IRC channel #systemd on libera.chat or Matrix channel
Stable branches with backported patches are available in the stable repo.
We have a security bug bounty program sponsored by the Sovereign Tech Fund hosted on YesWeHack
Repositories with distribution packages built from git main are available on OBS, and also repositories with packages built from the latest stable release