This is split out of #43058, but makes sense on its own: and makes
strip_tab_ansi() a lore more complete and correct.
ptybroker is going to use for the ansi sequence parser for other
purposes too, but making strip_tab_ansi() work better is enough of a
reason to have this I think.
ellipsize_mem() and align_string_mem() open coded the width of a character as
unichar_iswide(c) ? 2 : 1, but unichar_console_width() counts a tab as eight
cells. The two disagreed, and format-table.c uses both: it decides a field is too
wide with utf8_console_width(), then asks ellipsize() to fit it into that many
columns, and ellipsize() believed a tab was one cell. The ellipsized cell then
still overflowed the column, measured by the very function that had just asked
for it to be truncated. A table pinned to table_set_width(20) emitted lines of
25, 44 and 39 columns. align_string_mem() undercounted the same way and so padded
already overlong cells further.
Use unichar_console_width(), split out for this purpose in 5c0d4b5061. While at
it, let align_string_mem() decode with utf8_char_console_width() as well, so that
the second pass shares both the width model and the validity check with the
measuring pass, instead of keeping a more permissive decoder of its own around
for the next caller to trip over. Its "count invalid chars as 1" fallback now
skips the whole rejected sequence rather than one byte, so that the continuation
bytes are not charged a cell each of their own; the branch should be unreachable
anyway, since our caller only gets there after utf8_console_width() succeeded on
the same string.
Note that unichar_console_width() charges a tab a flat eight cells, while a
terminal advances to the next multiple of eight, so this is deliberately an upper
bound: the two agree only when the tab itself starts on a multiple of eight. A
per character helper cannot do better, because a tab's width is a function of the
column it starts in, and neither ellipsize_mem() nor align_string_mem() is told
what column its string starts at. Overestimating is the safe direction: it can
leave a column padded short, but it cannot overflow one.
ascii_ellipsize_mem() equates bytes with character cells, which holds for ASCII
except for the tab, so keep tabs off that fast path. Like the ANSI sequence case
just above it, such strings now take the generic path, which means they get the
Unicode ellipsis and one reserved cell rather than "..." and three, even in a non
UTF-8 locale.
So far strip_tab_ansi() was very superficial: it only focussed on OSC
and CSI sequences, and left the other ANSI sequences in touch.
Let's make it more powerful and port it to the more complete
AnsiSeqParser logic, and strip *all* ANSI sequences.
For the relevant usecases this seems like the better, safer choice,
since it's good to strip too much rather than too little, when
stripping ANSI sequences.
The backwards scan started at offset length-3, so the last position at which a
sequence can begin, length-2, was never examined. CSI sequences are at least
three bytes long and were thus unaffected, but two byte Fe sequences (ESC
followed by 0x40…0x5F) terminating the examined slice were missed.
ellipsize_mem() calls this to figure out whether a sequence ends exactly at the
current position, so that it can be skipped over, which is precisely the case
that was broken: such a sequence was instead counted as two visible cells and
copied through as text, and the string was ellipsized more aggressively than
requested. For example ellipsize("🐱🐱\x1bM🐱🐱\x1bM", 5, 0) returned a three
cell wide string rather than the five cells asked for.
udev_replace_whitespace() is documented to read at most 'len' bytes from
'str':
- the strspn() skip of leading whitespace stops at a non-space byte or
NUL, not at 'len'
- ata_id passes the space padded, non-NUL-terminated ATA IDENTIFY
model/serial/fw fields
- an all-blank field reads off the end of the 512-byte hd_driveid stack
struct
Capped the skip to 'len'; existing outputs are unchanged. Added a
regression test.
Like strspn(), but reads at most 'n' bytes from the input. strspn()
is bounded only by a non-matching byte or a NUL, so it over-reads a
buffer that is all matching bytes and not NUL terminated within 'n'.
Let's take inspiration from string_is_safe() and take a flags field that
allows fine tuning the validation.
Then port over all current users of either function to the new logic.
Note that this *does* change behaviour in various cases:
1. Generally: we'll now always accept the full UAPI.10 alphabet,
including the "~" and "^" characters. As far as I can see there's no
downside to this liberalization as none of the current consumers of
the two functions uses these characters for anything else.
2. systemd-analyze compare-version will now accept version strings with
"_" and "+" without complaining. I see no downside here, it just
normalizes these debugging tools, to make them accept what most our
other tools accept.
3. "bootctl link" will not accept empty version strings anymore
Which is a bugfix I guess.
4. vpick will now refuse "_" and "+" in version strings. It kinda
already did, because when parsing versions from filenames it uses "_"
and "+" as name, architecture and attempt counter separators. We now
systematically refuse it everywhere else in vpick too. This is hence
a clean-up.
Fixes: #28906
Replaces: #42815#41937
Let's bring the version string character set into a systematic order,
matching the order in which they appear and are defined in the UAPI.10
specification text.
This makes it easier to compare the relevant functions.
Allowing this apparently has been cargo-culted from my initial sysupdate
PR, but Claude and me could not find a single other software package
that uses "," as a character within version strings. Hence, let's remove
this, even though this is a compat breakage of a kind, in the hope
nobody notices. We can easily restore this if this later shows to be an
issue for people.
Validate UTF-8 characters against the caller-provided byte slice.
This stops truncated sequences from borrowing bytes past old_length.
Follow-up for e69cc9eb36
Lennart reminded me in [1] that we need to add assert() in functions
that do pointer access. For the simple `*p` pointer dereferences
we even have an automatic coccinelle script that ensures that as
part of the automatic code checks.
However for deref in the `p->` style this is not supported right
now and adding it to coccinelle is hard because its too slow for
this kind of check. So I created a (slightly messy) tree-sitter
python script to see how many asserts we are currently missing.
This commit is the result of running it over the `src/basic`
dir and fixing the flagged issues. I plan to tidy it up and
add it to the checks too but this is orthogonal to this commit.
[1] https://github.com/systemd/systemd/pull/42360#discussion_r3426964562
ellipsize_mem() scans backwards for ANSI escape sequences and calls
previous_ansi_sequence(s, t - s, ...) as t walks down toward s. When
t reaches s + 1 the helper is invoked with length == 1 and computes
'length - 2', which wraps to SIZE_MAX - 1.
Follow-up for cb558ab222
Transfer files might come and go, components might be enabled and
disabled. Patterns might change. Let's keep track of what we install, so
that we can automatically gc everything no longer owned by any enabled
transfer.
Whenever we are validating a string that shall appear in a filename
eventually we want to use filename_part_is_valid() rather than file
filename_is_valid(). Let's add explicit support for that to
string_is_safe(), since it's actually a really common case.
Split out of #41543.
These don't make too much sense on their own, but they also don't really
hurt. They are preparation for #41543, but in order to make things
either to review I split these four commits out, since they are not
directly part of what the PR shall achieve
This simply mirrors the same overflow check we already have in
strrepa(), in case someone passed us a sufficiently long string.
strrep() is currently used only in tests, so this is just hardening.
The updated parsing_hacks.h file uncovered a bunch of extra
things that the check-pointer-deref coccinelle script flags.
This commit fixes them to make the tree check-pointer-deref clean.
Add the needed assert changes to make the code clean
for the new check-pointer-deref script.
Note that for `_cmp()` functions its a bit of a question
if we want `assert()` or `POINTER_MAY_BE_NULL()` here. The
former is more correct but it adds a (small) runtime hit. If
we use POINTER_MAY_BE_NULL() it is compiled out but strictly
speaking it is wrong (so at least we would need a comment).
string_has_ansi_sequence() currently does this to look for ESC input:
t = memchr(s, 0x1B, ...)
So each iteration re-searches from the original start pointer. But if we
find an ESC byte that does *not* start a valid ANSI sequence (like "\x1B
", or an ESC at the end of the string), then ansi_sequence_length()
returns 0, and if that ESC is still in the search window, we will just
spin consuming 100% CPU forever.
Fix this by always advancing past rejected ESC bytes.
Since 6eabe9f2ff we generate sequences
with ":", hence we better also know how to strip them.
(Without this patch we'd strip simple ANSI colors, but not RGB ones that
use ":" syntax).
(While we are at it, also drop a duplicate "0" in the list of valid
chars)
Latest glibc uses _Generic to have strstr() and other functions return
const char* or char* based on whether the input is a const char* or a
char*. This causes build failures as we previously always expected a char*.
Let's fix the compilation failures and add our own macros similar to glibc's
to have string functions that return a mutable or const pointer depending on
the input.
As a wrapper for `find_line_startswith`, `find_line_after` search for
the exact line and return the pointer for the next line, or NULL if
missing.
`find_line` with search for the exact line and return the pointer to the
beginning of the line.
Signed-off-by: Alberto Planas <aplanas@suse.com>
Now that the necessary functions from log.h have been moved to macro.h,
we can stop including log.h in macro.h. This requires modifying source
files all over the tree to include log.h instead.
Admittedly, some of our glyphs _are_ special, e.g. "O=" for SPECIAL_GLYPH_TOUCH ;)
But we don't need this in the name. The very long names make some invocations
very wordy, e.g. special_glyph(SPECIAL_GLYPH_SLIGHTLY_UNHAPPY_SMILEY).
Also, I want to add GLYPH_SPACE, which is not special at all.
in strv_new() we have STRV_IGNORE for skipping over an argument in the
argument list. Let's add the same to strextend_with_separator():
strextend_with_separator(&x, "foo", POINTER_MAX, "bar");
will result in "foobar" being appended to "x". (POINTER_MAX Which is
different from NULL, which terminates the argument list).
This is useful for ternary op situations.
(We should probably get rid of STRV_IGNORE and just use POINTER_MAX
everywhere directly, but that's for another time.)
The functions are very similar, let's make them the same. If the first
argument to strextend() is NULL instead of extending a string we'll
allocate a fresh one and return that.
Yu Watanabe
Armaan Sandhu
Lennart Poettering
Syed Mohammed Nayyar
Luca Boccassi
Zbigniew Jędrzejewski-Szmek
Michael Vogt
Frantisek Sumsal
Chris Down
Lennart Poettering
Daan De Meyer
Mike Yuan
Alberto Planas
Zbigniew Jędrzejewski-Szmek