utf8,format-table: make the console width helpers agree with the UTF-8 validators (#43653)

systemd has two UTF-8 decoders, and the console width path uses the
permissive one.

`utf8_encoded_to_unichar()` performs no overlong check, no
`unichar_is_valid()` check, and still accepts the obsolete five and six
byte forms. Those checks live in the wrapper
`utf8_encoded_valid_unichar()`, which is what `utf8_is_valid()` and
`ellipsize_mem()` use. `utf8_char_console_width()` called the bare
decoder directly.

Checking both against `utf8_is_valid_n()` over all 4,244,897,280 NUL
free byte strings of length one to four, which is the entire UTF-8
encoding space:

| decoder | disagreements with `utf8_is_valid_n()` |
| --- | --- |
| `utf8_encoded_valid_unichar()` | 0 |
| `utf8_encoded_to_unichar()` | 8,835,838, always more permissive |

So `utf8_console_width()` returned a finite width for strings
`utf8_is_valid()` rejects. `format-table.c` measures a cell with
`utf8_console_width()` and, when it does not fit, hands it to
`ellipsize()`, which validates strictly and returns NULL. That is
reported as `-ENOMEM`:

```
lead surrogate    valid_utf8=0  utf8_console_width=25  ellipsize=NULL
                    -> table_format FAILS: Cannot allocate memory
```

A surrogate, overlong, noncharacter or five byte sequence in any cell
failed the whole table with a bogus out of memory error, and only once
that column was narrow enough to need truncation, so whether it happened
depended on the width of the user's terminal. At other widths the
invalid sequence was printed to the terminal instead. Invalid cells are
now refused consistently with `-EINVAL`, which is what the sequences
`utf8_encoded_to_unichar()` already rejected always produced.

The second commit fixes a related disagreement. `ellipsize_mem()` and
`align_string_mem()` open coded character width as `unichar_iswide(c) ?
2 : 1`, while `unichar_console_width()` counts a tab as eight cells.
`format-table.c` uses both: it finds a field too wide with
`utf8_console_width()`, then asks `ellipsize()` to fit that many
columns, and `ellipsize()` thought a tab was one cell, so the result
still overflowed the column that the measurement had just asked it to
fit. A table pinned to `table_set_width(20)` emitted lines of 25, 44 and
39 columns. This uses `unichar_console_width()`, which was split out for
exactly this purpose in 5c0d4b5061.

Both commits add tests that fail before the change and pass after. The
full unit suite is otherwise unaffected.

src/shared/format-table.c

@@ -2079,18 +2079,23 @@ static char* align_string_mem(const char *str, const char *url, size_t new_lengt
         } else
                 clickable_length = old_length;
 
-        /* Determine current width on screen */
+        /* Determine current width on screen. Use the same helper the measuring pass uses, so that this
+         * function cannot keep a second, more permissive idea of both validity and width of its own.
+         * Note that our caller only gets here after utf8_console_width() succeeded on the very same
+         * string, so the invalid case below should not be reachable in practice. */
         p = str;
         while (p < str + old_length) {
-                char32_t c;
-
-                if (utf8_encoded_to_unichar(p, &c) < 0) {
-                        p++, w++; /* count invalid chars as 1 */
+                int n = utf8_char_console_width(p);
+                if (n < 0) {
+                        /* Count an invalid sequence as one character of width one. Skip all of it, so
+                         * that its continuation bytes are not charged a cell each of their own. */
+                        p = utf8_next_char(p);
+                        w++;
                         continue;
                 }
 
                 p = utf8_next_char(p);
-                w += unichar_iswide(c) ? 2 : 1;
+                w += n;
         }
 
         /* Already wider than the target, if so, don't do anything */