The color list is extended by searching, suggestions from llms, etc.
In particular, I included "-ish" variants of common colors: "greenish",
"reddish", etc. They are easy to identify and I think quite distinct from
the base versions.
For weather words, add more stuff. Some are not commonly used, but mostly
easy to spell. In the worst case, people learn more about dreary weather.
For worlds, add a bunch of solar system moons, asteroids, trans-Neptunian
objects, and mythological realms.
This raises the number of combinations from 68'880 to appox. 2'300'000,
making collisions and repeats less likely.
The comment was wrong. The average word is about 6 characters, so in
64 iterations, the probability of never landing on a word boundary is
~(1 - 1/7)**64, about 0.01%.
hostname_pick_word() used to seek around in the wordlist file and read
individual lines with read_line()/read_stripped_line(). Instead, slurp the
whole file into memory use offset arithmetic. The wordlists are small, so
reading them entirely is faster and there is really no reason to do the
complicated thing. The new code is 15% faster (the slow parts are the
hashing and chaseat, they actually dominate over the reading of the data).
The "raced with truncation" case is gone, since the buffer is a consistent
snapshot of the file.
The random offsets derived from the machine ID are unchanged, so the
picked words stay the same.
112cfb1814 tried to improve handling of
paths when root is specified. It said:
> To avoid answering what a relative path means when called with a root
> prefix path_strv_canonicalize is now path_strv_canonicalize_absolute
> and only considers absolute paths. Fortunately all users of already call
> path_strv_canonicalize with a list of absolute paths.
Of course, other users were added later and for example now
SYSTEMD_HOSTNAME_WORDLIST_PATH=build/hostname-wordlist/ ... doesn't
work as expected. If root is not specified, there is really no reason
to ignore relative paths. They work relative to CWD per the usual convention.
It's up to the user to not use relative paths in a context where they
should not be used. When root is specified, continue to ignore the relative
paths, since there is no clear interpreation what they mean, but emit
a debug message.
This is useful to check if how fast the code is (painfully slow) and whether
worlds indeed are selected with a flat distribution probability (they are).
The code is slow because the hashing and because chaseat is exteremely
inefficient :(
Also add a note that the function does not support multiple groups. I
think that's fair, it doesn't make sense to generalize the functionality
when we're unlikely to ever use that. (Verb options are only used by
udevadm right now.)
Let's treat unset SYSTEMD_INVOKED_AS and SYSTEMD_INVOKED_AS='' the same.
I encountered this when testing various --help outputs. It is often
more convenient to override a variable with VAR= instead of removing it,
in particular on the commandline. But we'd then print help with the
program name missing. We look at the variable in two places, so adjust
both as in $subject.
All programs print their help through the COMMAND machinery now:
- the VERB_COMMON_HELP() and VERB_COMMON_HELP_HIDDEN() macros which
wired the help verb to a hand-written help() implementation are
unused — everything uses the VERB_COMMON_HELP_AUTO*() family,
- the verbs help table is only built internally, so the function
becomes static, as the TODO promised,
- help_abstract() and help_man_page_reference() are unused — the
generated help renders the abstract via print_wrapped() and the man
page links via print_man_links().
The hand-written help() functions were the only users of the
namespace+group help table builder, and they are all gone. The one
remaining caller in the verbs machinery is ported to
options_find_namespace() + options_get_help_table_group(), which
build the identical table.
Also drop OPTION_COMMON_INTROSPECT_CLI_ENTRY: it has no users, and
references the no-longer-existing _OPTION_ENTRY() macro, so it could
not even be used.
The variadic pair-wise variant was only used by the hand-written help()
functions to synchronize their verb and option tables. Those are all
gone, and the verbs machinery uses table_sync_all_column_widths().
The last five hand-written help() functions are replaced by COMMAND()
descriptions: test-homectl-prompts, test-libudev,
test-modem-manager-mock, test-chase-manual, and test-journal-append
(which open-coded the help inside the option parsing loop). None of the
binaries are installed, so no man pages are referenced and they are not
added to the introspection smoke test.
The custom help() is replaced by a COMMAND() description, matching the
other report-sign services. The abstract is now wrapped at 80 columns
in the help output.
Negative return values, e.g. from option parsing, were propagated verbatim
as the exit status of main() and truncated to meaningless values (-EINVAL
became 234). The positive-failure variant is needed because the binary may
exit with EXIT_TEST_SKIP.
Also propagate EXIT_TEST_SKIP instead of suppressing it.
Unfortunately udevadm is special, so some more infra extensions turned
out to be necessary.
Retroactively, we might want to apply some of the new functionality in
other places too, but I'm leaving that for later.
systemd-udevd is the same binary as udevadm, so the introspection data
describes both commands, from either name. The COMMAND intentionally
owns no verbs — all of udevadm's verbs are defined in udevadm.c right
after its own COMMAND, so that they are attributed to it.
The -V/--version output (GIT_VERSION) is preserved.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-wait" option namespace. The verb's one-line
description from the abstract is retained, I think it is good enough
but shorter.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-verify" option namespace.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-trigger" option namespace.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-test-builtin" option namespace. The list of
built-in commands is appended after the generic help. The man page
reference is now printed before the command list. It'd be somewhat
nicer to have it _after_, but that'd require a lot of plumbing to
split the help generated in two and just doesn't seem worth effort.
The list of commands is now printed to stdout (it went to stderr before),
arguably a backportable bugfix.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-test" option namespace.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-settle" option namespace.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-monitor" option namespace.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-control" option namespace. The synopsis
makes use of the new nulstr-argspec functionality:
> udevadm [OPTION…] lock [OPTION…] COMMAND
> udevadm [OPTION…] lock [OPTION…] --print
The arg_no_pager variable is lifted to udevadm, so that it can be
referenced in the command structure. It's only used for 'udevadm info',
but that's OK. We can later generalize it if wanted.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in udevadm.c
and the "udevadm-info" option namespace. --help now goes through the
pager, like other commands with --no-pager.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in udevadm.c
and the "udevadm-control" option namespace. The new VERB_DEPRECATED flag
is used.
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-control" option namespace. The synopsis
is now "udevadm [OPTION…] control OPTION…".
The custom help() is replaced by the common verb help, which derives the
synopsis, abstract, and option list from the verb declaration in
udevadm.c and the "udevadm-cat" option namespace.
The verbs declare their subcommand option namespaces, so the
introspection data describes the options of every subcommand, nested in
the verb objects.
The bare version number output (a dracut compatibility requirement) is
preserved for --version/-V and the "version" verb.
The "lock" verb's argument specification is simplified to "COMMAND" --
other verbs do not mention their options in the argspec either, and the
subcommand help states where the options may go.
Previously, verbs were skipped in --help automatically when their
.help field was nil. But this doesn't work in two ways:
- in introspection data we want to show the option, but marked as
deprecated
- in udevadm (and maybe other programs with verbs with own options in
the future), we need to show help for the verb.
So add an explicit flag to mark verbs as deprecated and skip them in
help even if .help is set. If there's a footer, print it highlighted
in red automatically.
A helper to print the help of a single verb, derived entirely from the
data in the SYSTEMD_VERBS and SYSTEMD_OPTIONS sections: the synopsis is
built from the verb name and argspec, the one-line verb help serves as
the abstract, the options come from the verb's option namespace, and the
man page links from the command the verb belongs to. A flag allows
suppressing the man page links for callers which append custom sections.
Like the command-level CommandDescription.argspec, a verb's argspec is now
a nulstr, so that a verb may declare multiple alternative usage lines. The
verbs help table renders one synopsis line per nulstr entry in the
synopsis cell.
All verb argument specifications must gain an explicit NUL suffix. This
is a lot of churn, but I think this needs to be done. For now, there'll
be only one user of the new functionality (i.e. a verb with more than
one argspec), but I think we'll eventually want to use this in other
places too.
Verbs like udevadm's subcommands parse their own options from a dedicated
option namespace. Allow the namespace to be declared in the verb
definition (via the new VERB_SCOPE_NS* macros), and report the options of
that namespace in the verb's CLI introspection object.
This is conformant with the CLI-Introspection Specification: verbs are
represented as nested command objects, which may carry their own "options"
array just like the top-level command object.
systemd-shutdown parsed a bunch of options, but had no way to print
them. Add the standard --help and --version options and the CLI
introspection.
parse_argv() is moved before the PID 1 check, so that the new options
also work when the binary is invoked manually. As PID 1 we can never
return from main() (the kernel would panic), so in that case parse
errors still freeze(), and the help output is followed by a freeze()
too. Ctrl-Alt-Delete and console reopening are re-enabled before
freezing, so that the freeze can be escaped through the kernel and the
error is visible, as before.
The --help/--version/introspect-cli option names are already listed in
the kernel command line exclusion list in proc-cmdline.c.
--introspect-cli follows the same deferred pattern as --help: parse_argv()
only records the action, and it is executed in the same place where
ACTION_HELP and ACTION_VERSION are handled.
The help is looked up by explicit name: in the multicall configuration
the systemd and systemd-executor commands share one binary, so the
"first COMMAND" default would depend on link order.
The "This program takes no positional arguments." line is dropped from
the help output — the default "[OPTION…]" synopsis already says so.
(And also, this is not quite true. It does take positional args when
called as init.)
This converts systemd-homed, -hostnamed, -importd, -localed, -logind,
-machined, -networkd, -portebled, -resolved, -sysupdated, -timedated,
and -timesyncd.
The service name and description move from service_parse_argv() arguments
into a COMMAND() description that each daemon now defines. The
service_parse_argv() macro captures the calling binary's verb section,
where that COMMAND is found, while the options are still defined and
parsed in libshared, in the new "service" option namespace that each
COMMAND references. (The named namespace makes sure that options defined
by other translation units in libshared can never leak into the
services' parsers.)
The supported features are declared via option group filtering: each
COMMAND lists "Options" plus "Bus introspection" and/or "Runtime scope".
Unsupported options are now rejected by the parser as unknown options
(and are not shown in --help), replacing the runtime "This service does
not support …" errors.
The help output changes slightly: the "Bus introspection" and "Runtime
scope" option groups now get their own headings, and some descriptions
were reworded to the imperative form used elsewhere.
Standard parse_argv() func is added. --version is now accepted
(previously only the help forms were recognized, via argv_looks_like_help()).
The synopsis changes from "systemd-integritysetup COMMAND ..." to the
standard "systemd-integritysetup [OPTION…] COMMAND …", and the help output
gains an Options section. "help" as the first argument keeps working via
the hidden common help verb.
Previously, we'd just print help and exit with success if no arguments
were given. This seems unexpected and wrong. The program now prints the
usual error message:
Command verb required (one of attach, detach, help).
This time, various commands with non-trivial requirements are handled.
So the series is shorter, but has a pattern of feature additions and
then those new features being used. The way that --help output is
formatted is changed a bit in various places. Those changes are
described in individual commits.
22 'static int help()' functions left after this.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added.
Options are now in two groups: "Common options" and Mount options". In
--help, for systemd-mount, both groups are shown. For systemd-umount, only
the first. This changes the behaviour a bit: previously, both groups were
shown together as one list. I think the new behaviour is actually a bit
better, because it makes it easier for the user to figure out that the
commands share common options and one of the commands has its private
second group. But if people think that the previous behaviour should
be restored, we could add a display options to squish the groups back
together.
--help now goes through the pager, like other commands with --no-pager.
So far, option groups were only used for display: when parsing options,
the group markers were ignored and all options within a namespace were
considered as potential matches. The group was reported in introspection
data and used as section seperator in help display. To allow a single
option namespace to be used for two commands, pipe through a nulstr with
option group names to look at when parsing options. The field is set in
the Command description and used in various option-handling contexts:
- when parsing and looking for an option, do not consider the entries
that are in the group that is not listed in the nulstr,
- when listing options in help or introspection, skip over unlisted
groups.
The verbs_find_verb and verbs_find_command helpers are adjusted to have
a consistent interface.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. --help now goes through the pager, like other
commands with --no-pager.
Also, previously, -h/--help/--version were listed in the "Commands"
section. This was cute, but recreating this in the new scheme would
be complicated. Also, it seems unnecessary. --help/--version and some
other commands terminate the program in all binaries, so they could
be considered "commands" in all cases. But we printed them together
with verbs only in a few cases. So I think it's fine to move them
back to the "Options" section. This makes the description of options
much more consistent between different programs.
This allows commands like systemd-sysext + systemd-confext to be handled
without duplicating entries in the verbs table. The basic idea is to have
the following layout in the SYSTEMD_VERBS section:
VERB_COMMAND_MARKER → "systemd-sysext", .flags = COMMAND_VERBS_SHARED
VERB_COMMAND_MARKER → "systemd-confext", .flags = COMMAND_VERBS_SHARED
VERB("status"),
VERB("merge"),
...
And then in places where we want to work on the verbs, we skip over the
verb entries with VERB_COMMAND_MARKER and COMMAND_VERBS_SHARED flag set.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The note about invocation as 'installkernel' is
kept as the footer. Verb and option column widths are kept separate, since
the verb table is very wide. --help now goes through the pager, like other
commands with --no-pager.
A separate COMMAND block is included for installkernel, but it is only
used for introspection data.
The custom halt_help() and shutdown_help() are replaced by COMMAND()
descriptions (one per command name, halt/poweroff/reboot sharing the
"halt" option namespace), and --introspect-cli is added to both parsers.
halt/poweroff/reboot now each reference their own man page name.
In review it was raised that the file is often synthesized and that this
synthetization might not include all the fields, let's allow the more
recently added fields to be missing to avoid a regression in
compatibility.
As discussed in the PR, those fields can go down, so they cannot be advertised
as type COUNTER. But the type is shared for the whole family, so we need to
create a new family to keep the other ones as COUNTER and add the new ones
as GAUGE.
kernel/sched/cputime.c says above get_cpu_idle_time_us() and get_cpu_iowait_time_us():
> Note that this is partially broken due to the counter of iowait tasks that
> can be remotely updated without any synchronization. Therefore it is possible
> to observe backward values within two consecutive reads.
steal is apparently also unreliable because it is reported by the hyperviser.
So let's be cautious and report it as gauge too.
Extend the io.systemd.Basic metric families with host-level runtime
metrics: cumulative per-state CPU seconds, memory in use (MemTotal minus
MemAvailable), swap in use, the kernel's 10s pressure stall averages
(cpu/memory/io x some/full). Partitions and stacked devices are excluded
from the disk counters, since their traffic is already accounted for in
the parent device.
Also extend the TEST-74 report subtest with checks for the new metrics.
Network metrics are not added, because the service runs with
PrivateNetwork=yes, and it doesn't make sense to report stats for 'lo'.
After some back-and-forth, I settled on "CPUUsage". We use this scheme of
capitalization in other metric family names: "IPv4Public", "IPv6Public",
IPv4AddressState", "IPv6AddressState", "OSRelease" "SMBIOS" "BootID",
"CPUsOnline", "MachineID", "MachineInfo", "OSRelease",
"SMBIOS.SystemManufacturer", "TPM2.Manufacturer", "TPM2.VendorString",
"IOReadBytes", "IOReadOperations", "CGroup". The exception is
"io.systemd.cgroup.CpuUsage". I think we should treat that one as a
mistake and consider changing it to conform.
In practice I think this will ever only be used with ansi_highlight_red(),
but it's cleaner to add this generic extension rather than e.g. a boolean
highlight_footer_red.
The options is passed through via command_print_help_full(), not through
the COMMAND structure, which means that commands with 'help' verb are not
covered by this. The red footer highligting is used for legacy commands,
which don't have verbs, so I think this is OK. Making it generic and
covering 'help' verbs would require more refactoring, and I think it
doesn't make sense to add it until it's actually needed.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The generic help-verb check in the test is
filtered out, since systemctl's "help" verb shows unit manuals and fails
without arguments.
Almost all programs are simple and have only a single command. When
printing options, let's just use the first COMMAND block found by default.
This means we can drop the explicit program-name argument and reduce
copy&paste boilerplate.
The kernel reports guest usage in guest/guest-nice and also includes it in
user/nice. By adding the guest and guest-nice values, we were counting
those ticks twice.
C.f. https://gitlab.com/procps-ng/procps/-/blob/master/library/stat.c?ref_type=heads#L612-615
> note: we exclude guest tics from xtot since ...
> 'user' already includes 'guest'
> 'nice' already includes 'gnice'
> (see linux sources: ./kernel/sched/cputime.c, kcpustat_cpu_fetch)
Rewrite procfs_cpu_get_usage() to use procfs_cpu_get_ticks() and fix the
accounting error.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The referenced systemd-report-cgroup(8) man
page does not exist, so the reference is dropped.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The abstract gets a trailing period. There is
no man page for this helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this internal helper,
so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this internal helper,
so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this internal helper,
so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this internal helper,
so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The abstract is changed to the imperative.
There is no man page for this udev helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The abstract is changed to the imperative.
There is no man page for this udev helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this udev helper, so
none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. There is no man page for this udev helper, so
none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The help output had no abstract, so one is
added. There is no man page for this udev helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The help output had no abstract, so one is
added. There is no man page for this udev helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The help output had no abstract, so one is
added. There is no man page for this udev helper, so none is referenced.
The custom help() is replaced by a COMMAND() description, and
--introspect-cli is added. The tool has no man page of its own, so
iocost.conf(5), which describes it, is referenced. The test now also looks
for commands in /usr/lib/udev, where the udev helpers are installed.
The .abstract field is always set, so it is fairly unambiguous to
recognize, and the value is usually long, so it's better to be brief.
It was only explicitly labelled in those two places.
