We use symbols provided by unistd.h without including it. E.g.
open(), close(), read(), write(), access(), symlink(), unlink(), rmdir(),
fsync(), syncfs(), lseek(), ftruncate(), fchown(), dup2(), pipe2(),
getuid(), getgid(), gettid(), getppid(), pipe2(), execv(), _exit(),
environ, STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO, F_OK, and their
friends and variants, so on.
Currently, unistd.h is indirectly included mainly in the following two paths:
- through missing_syscall.h, which is planned to covert to .c file.
- through signal.h -> bits/sigstksz.h, which is new since glibc-2.34.
Note, signal.h is included by sd-eevent.h. So, many source files
indirectly include unistd.h if newer glibc is used.
Currently, our baseline on glibc is 2.31. We need to support glibc older
than 2.34, but unfortunately, we do not have any CI environments with
such old glibc. CIFuzz uses glibc-2.31, but it builds only fuzzers, and
many files are even not compiled.
To make sure everything still compiles, we add a preliminary include
of forward.h to tests.h to make sure it is included in every test source
file. We'll clean up the tests.h includes in a later commit.
We also add a <errno.h> include to errno-list.h to keep test-errno-list.c
compiling. It'll be removed again when we clean up includes in src/basic.
Split out of #37344.
If the build directory is outside of the source tree, ASSERT_XYZ()
triggers the following warning:
===
[20/569] Compiling C object test-hashmap.p/meson-generated_.._src_test_test-hashmap-ordered.c.o
In file included from ../../home/watanabe/git/systemd/src/basic/macro.h:399,
from ../../home/watanabe/git/systemd/src/basic/alloc-util.h:10,
from src/test/test-hashmap-ordered.c:5:
src/test/test-hashmap-ordered.c: In function ‘test_ordered_hashmap_get’:
../../home/watanabe/git/systemd/src/basic/log.h:216:27: warning: offset ‘32’ outside bounds of constant string [-Warray-bounds=]
216 | ? log_internal(_level, _e, PROJECT_FILE, __LINE__, __func__, __VA_ARGS__) \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../home/watanabe/git/systemd/src/basic/log.h:238:24: note: in expansion of macro ‘log_full_errno_zerook’
238 | (void) log_full_errno_zerook(level, 0, fmt, ##__VA_ARGS__); \
| ^~~~~~~~~~~~~~~~~~~~~
../../home/watanabe/git/systemd/src/basic/log.h:248:28: note: in expansion of macro ‘log_full’
248 | #define log_error(...) log_full(LOG_ERR, __VA_ARGS__)
| ^~~~~~~~
../../home/watanabe/git/systemd/src/shared/tests.h:251:25: note: in expansion of macro ‘log_error’
251 | log_error("%s:%i: Assertion failed: expected \"%s\" to be NULL, but \"%p\" != NULL", \
| ^~~~~~~~~
src/test/test-hashmap-ordered.c:614:9: note: in expansion of macro ‘ASSERT_NULL’
614 | ASSERT_NULL(r);
| ^~~~~~~~~~~
===
Currently, hashmap_dump_sorted sorts by key and then returns the values
in order sorted by key. This commit adds another helper that does the
same but returns the sorted keys instead
This converts to TEST macro in less trivial cases. This is mostly
due to having an intro or outro before/after the actual tests.
Some notable changes:
- add a "test" to make sure the hashmap and ordered_hashmap tests
from different compilation units are actually run in test-hashmap.c
- make root arg a global var in test-install-root.c
- slightly rework an EFI specific test in test-proc-cmdline.c
- usage of saved_argv/saved_argc in test-process-util.c
- splitting test-rlimit-util.c into several tests
- moving the hwdb open check into intro in test-sd-hwdb.c
- condense several "tests" into one in test-udev-util.c
This also makes path_compare() may return arbitrary integer as it now
simply pass the result of strcmp() or memcmp().
This changes the behavior of path_extract_filename/directory() when
e.g. "/." or "/./" are input. But the change should be desired.
So far, we'd use hashmap_free_free to free both keys and values along with
the hashmap. I think it's better to make this more encapsulated: in this variant
the way contents are freed can be decided when the hashmap is created, and
users of the hashmap can always use hashmap_free.
The point here is to compare speed of hashmap_destroy with free and a different
freeing function, to the implementation details of hashmap_clear can be
evaluated.
Results:
current code:
/* test_hashmap_free (slow, 1048576 entries) */
string_hash_ops test took 2.494499s
custom_free_hash_ops test took 2.640449s
string_hash_ops test took 2.287734s
custom_free_hash_ops test took 2.557632s
string_hash_ops test took 2.299791s
custom_free_hash_ops test took 2.586975s
string_hash_ops test took 2.314099s
custom_free_hash_ops test took 2.589327s
string_hash_ops test took 2.319137s
custom_free_hash_ops test took 2.584038s
code with a patch which restores the "fast path" using:
for (idx = skip_free_buckets(h, 0); idx != IDX_NIL; idx = skip_free_buckets(h, idx + 1))
in the case where both free_key and free_value are either free or NULL:
/* test_hashmap_free (slow, 1048576 entries) */
string_hash_ops test took 2.347013s
custom_free_hash_ops test took 2.585104s
string_hash_ops test took 2.311583s
custom_free_hash_ops test took 2.578388s
string_hash_ops test took 2.283658s
custom_free_hash_ops test took 2.621675s
string_hash_ops test took 2.334675s
custom_free_hash_ops test took 2.601568s
So the test is noisy, but there clearly is no significant difference with the
"fast path" restored. I'm surprised by this, but it shows that the current
"safe" implementation does not cause a performance loss.
When the code is compiled with optimization, those times are significantly
lower (e.g. 1.1s and 1.4s), but again, there is no difference with the "fast
path" restored.
The difference between string_hash_ops and custom_free_hash_ops is the
additional cost of global modification and the extra function call.
This part of the copyright blurb stems from the GPL use recommendations:
https://www.gnu.org/licenses/gpl-howto.en.html
The concept appears to originate in times where version control was per
file, instead of per tree, and was a way to glue the files together.
Ultimately, we nowadays don't live in that world anymore, and this
information is entirely useless anyway, as people are very welcome to
copy these files into any projects they like, and they shouldn't have to
change bits that are part of our copyright header for that.
hence, let's just get rid of this old cruft, and shorten our codebase a
bit.
Files which are installed as-is (any .service and other unit files, .conf
files, .policy files, etc), are left as is. My assumption is that SPDX
identifiers are not yet that well known, so it's better to retain the
extended header to avoid any doubt.
I also kept any copyright lines. We can probably remove them, but it'd nice to
obtain explicit acks from all involved authors before doing that.
There are more than enough calls doing string manipulations to deserve
its own files, hence do something about it.
This patch also sorts the #include blocks of all files that needed to be
updated, according to the sorting suggestions from CODING_STYLE. Since
pretty much every file needs our string manipulation functions this
effectively means that most files have sorted #include blocks now.
Also touches a few unrelated include files.
The purpose of testing with the crippled hash function is to cover
the otherwise very unlikely codepath in bucket_calculate_dib() where
it has to fall back to recomputing the hash value.
This unlikely path was not covered by test-hashmap anymore after
57217c8f "test: hashmap - cripple the hash function by truncating the
input rather than the output".
Restore the test coverage by increasing the number of entries in the test.
The number was determined empirically by checking with lcov.
All our hash functions are based on siphash24(), factor out
siphash_init() and siphash24_finalize() and pass the siphash
state to the hash functions rather than the hash key.
This simplifies the hash functions, and in particular makes
composition simpler as calling siphash24_compress() repeatedly
on separate chunks of input has the same effect as first
concatenating the input and then calling siphash23_compress()
on the result.
The reason for the crippled hash function is to reduce the distribution
of the hash function, do this by truncating the domain rather than the
range. This does introduce a change in behavoir as the range is no longer
contiguous, which greatly reduces collisions.
This is needed as a follow-up patch will no longer allow individual hash
functions to alter the output directly.
Currently, the HASHMAP iterators stop at the first NULL entry in a
hashmap. This is non-obvious and breaks users like sd-device, which
legitimately store NULL values in a hashmap.
Fix all the iterators by taking a pointer to the value storage, instead of
returning it. The iterators now return a boolean that tells whether the
end of the list was reached.
Current users of HASHMAP_FOREACH() are *NOT* changed to explicitly check
for NULL. If it turns out, there were users that inserted NULL into
hashmaps, but didn't properly check for it during iteration, then we
really want to find those and fix them.
Yu Watanabe
Daan De Meyer
Integral
Matteo Croce
Ivan Kruglov
Adrian Vovk
David Tardon
Jan Janssen
Zbigniew Jędrzejewski-Szmek
Lennart Poettering
Michal Schmidt
Tom Gundersen
David Herrmann
Thomas Hindoe Paaboel Andersen
Martin Pitt