It is unused and rightly so. Users of the hashmap API should
not care about the idx values or any other Iterator internals.
_IDX_ITERATOR_FIRST in hashmap.h is an exception. It is needed
for ITERATOR_FIRST.
The comparison function we use for qsorting paths is overly indifferent.
Consider these 3 paths for sorting:
/foo
/bar
/foo/foo
qsort() may compare:
"/foo" with "/bar" => 0, indifference
"/bar" with "/foo/foo" => 0, indifference
and assume transitively that "/foo" and "/foo/foo" are also indifferent.
But this is wrong, we want "/foo" sorted before "/foo/foo".
The comparison function must be transitive.
Use path_compare(), which behaves properly.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1184016
Commit 668c965af "journal: skipping of exhausted journal files is bad if
direction changed" fixed a correctness issue, but it also significantly
limited the cases where the optimization that skips exhausted journal
files could apply.
As a result, some journalctl queries are much slower in v219 than in v218.
(e.g. queries where a "--since" cutoff should have quickly eliminated
older journal files from consideration, but didn't.)
If already in the initial iteration find_location_with_matches() finds
no entry, the journal file's location is not updated. This is fine,
except that:
- We must update at least f->last_direction. The optimization relies on
it. Let's separate that from journal_file_save_location() and update
it immediately after the direction checks.
- The optimization was conditional on "f->current_offset > 0", but it
would always be 0 in this scenario. This check is unnecessary for the
optimization.
Make sure we don't inspect sun_path of unnamed sockets.
Since we cannot know if two unnamed sockets' adresses refer to the same
socket, just return false.
When dbus.socket is updated like this:
-ListenStream=/var/run/dbus/system_bus_socket
+ListenStream=/run/dbus/system_bus_socket
... and daemon-reload is performed, bad things happen.
During deserialization systemd does not recognize that the two paths
refer to the same named socket and replaces the socket file with a new
one. As a result, applications hang when they try talking to dbus.
Fix this by finding a match not only when the path names are equal, but
also when they point to the same inode.
In socket_address_equal() it is necessary to move the address size
comparison into the abstract sockets branch. For path name sockets the
comparison must not be done and for other families it is redundant
(their sizes are constant and checked by socket_address_verify()).
FIFOs and special files can also have multiple pathnames, so compare the
inodes for them as well. Note that previously the pathname checks used
streq_ptr(), but the paths cannot be NULL.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1186018
The output of gnutls_certificate_verification_status_print() needs to be
freed.
Noticed this while staring at verify_cert_authorized() to see what could
possibly confuse gcc5 on armv7hl to segfault during compilation.
EOF is meaningless if the direction of iteration changes.
Move the EOF optimization under the direction check.
This fixes test-journal-interleaving for me.
Thanks to Filipe Brandenburger for telling me about the failure.
next_with_matches() is odd in that its "unit64_t *offset" parameter is
both input and output. In other it's purely for output.
The function is called from two places in next_beyond_location(). In
both of them "&cp" is used as the argument and in both cases cp is
guaranteed to equal f->current_offset.
Let's just have next_with_matches() ignore "*offset" on input and
operate with f->current_offset.
I did not investigate why it is, but it makes my usual benchmark run
reproducibly faster:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m4.032s
user 0m3.896s
sys 0m0.135s
(Compare to preceding commit, where real was 4.4s.)
I accidentally broke the detection of duplicate entries in 7943f42275
"journal: optimize iteration by returning previously found candidate
entry".
When we have a known location of a candidate entry, we must not return
from next_beyond_location() immediately. We must go through the
duplicates detection to make sure the candidate differs from the
already iterated entry.
This fix slows down iteration a bit, but it's still faster than it
was before the rework.
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m4.448s
user 0m4.298s
sys 0m0.149s
(Compare with results from commit 7943f42275, where real was 5.3s before
the rework.)
In next_beyond_location() when the JournalFile's location type is
LOCATION_SEEK, it means there's nothing to do, because we already have
the location of the candidate entry. Do an early return. Note that now
next_beyond_location() does not anymore guarantee on return that the
entry is mapped, but previous patches made sure the caller does not
care.
This optimization is at least as good as "journal: optimize iteration:
skip files that cannot improve current candidate entry" was.
Timing results on my workstation, using:
$ time ./journalctl -q --since=2014-06-01 --until=2014-07-01 > /dev/null
Before "Revert "journal: optimize iteration: skip files that cannot
improve current candidate entry":
real 0m5.349s
user 0m5.166s
sys 0m0.181s
Now:
real 0m3.901s
user 0m3.724s
sys 0m0.176s
If from a previous iteration we know we are at the end of a journal
file, don't bother looking into the file again. This is complicated by
the fact that the EOF does not have to be permanent (think of
"journalctl -f"). So we also check if the number of entries in the
journal file changed.
This optimization has a similar effect as "journal: optimize iteration:
skip whole files behind current location" had.
set_location() is called from real_journal_next() when a winning entry
has been picked from among the candidates in journal files.
The location type is always set to LOCATION_DISCRETE. No need to pass
it as a parameter.
The per-JournalFile location information is already updated at this
point. No need for having the direction and offset here.
In next_beyond_location() when we find a candidate entry in a journal
file, save its location information in struct JournalFile.
The purpose of remembering the locations of candidate entries is to be
able to save work in the next iteration. This patch does only the
remembering part.
LOCATION_SEEK means the location identifies a candidate entry.
When a winner is picked from among candidates, it becomes
LOCATION_DISCRETE.
LOCATION_TAIL here signifies we've iterated the file to the end (or the
beginning in the case of reversed direction).
try_context() is such a hot path that the hashmap lookup is expensive.
The number of contexts is small - it is the number of object types.
Using a hashmap is overkill. A plain array will do.
Before:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m9.445s
user 0m9.228s
sys 0m0.213s
After:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m5.438s
user 0m5.266s
sys 0m0.170s
Note that numbers 0 and -1 are both replaced with OBJECT_UNUSED,
because they are treated the same everywhere (e.g. type_to_context()
translates them both to 0).
If type==0 and a non-NULL object were given as arguments to
journal_file_hmac_put_object(), its object type check would fail and it
would return -EBADMSG.
All existing callers use either a positive type or -1. Still, for
behavior consistency with journal_file_move_to_object() let's allow
type 0 to pass.
The only user is sd_journal_enumerate_unique() and, as explained in
the previous commit (fed67c38e3 "journal: map objects to context set by
caller, not by actual object type"), the use of them there is now
superfluous. Let's remove them.
This reverts major parts of commits:
ae97089d49 journal: fix access to munmapped memory in
sd_journal_enumerate_unique
06cc69d44c sd-journal: fix sd_journal_enumerate_unique skipping values
Tested with an "--enable-debug" build and "journalctl --list-boots".
It gives the expected number of results. Additionally, if I then revert
the previous commit ("journal: map objects to context set by caller, not
to actual object type"), it crashes with SIGSEGV, as expected.
When the caller of journal_file_move_to_object() specifies type==0,
the object header is at first mapped in context 0. Then after the header
is checked, the whole object is mapped in a context determined by
the actual object type (which is not even range-checked using
type_to_context()). This looks wrong. It should map in the
caller-specified context.
An old comment in sd_journal_enumerate_unique() supports this view:
/* We do not use the type context here, but 0 instead,
* so that we can look at this data object at the same
* time as one on another file */
Clearly the expectation was that the data object will remain mapped
in context 0 without being pushed away by mapping other objects in
context OBJECT_DATA.
I suspect that this was the real bug that got fixed by ae97089d49
"journal: fix access to munmapped memory in sd_journal_enumerate_unique".
In other words, journal_file_object_keep/release are superfluous after
applying this patch.
This is useful for exposing unsafe access to mmapped objects after
the context that they were mapped in was already moved.
For example:
journal_file_move_to_object(f1, OBJECT_DATA, p1, &o1);
journal_file_move_to_object(f2, OBJECT_DATA, p2, &o2);
t = o1->object.type; /* this usually works, but is unsafe */
There will be more debugging options later.
--enable-debug will enable them all.
--enable-debug=hashmap will enable only hashmap debugging.
Also rename the C #define to ENABLE_DEBUG_* pattern.
Suppose that while iterating we have already looked into a journal file
and got a candidate for the next entry. And we are considering to look
into another journal file because it may contain an entry that is nearer
to the current location than the candidate.
We should skip the whole journal file if we can tell by looking at its
header that none of its entries can precede the candidate.
Before:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m20.518s
user 0m19.989s
sys 0m0.328s
After:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m9.445s
user 0m9.228s
sys 0m0.213s
Interleaving of entries from many journal files is expensive. But there
is room for optimization.
We can skip looking into journal files whose entries all lie before the
current iterating location. We can tell if that's the case from looking
at the journal file header. This saves a huge amount of work if one has
many of mostly not interleaved journal files.
On my workstation with 90 journal files in /var/log/journal/ID/
totalling 3.4 GB I get these results:
Before:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 5m54.258s
user 2m4.263s
sys 3m48.965s
After:
$ time ./journalctl --since=2014-06-01 --until=2014-07-01 > /dev/null
real 0m20.518s
user 0m19.989s
sys 0m0.328s
The high "sys" time in the original was caused by putting more stress on
the mmap-cache than it could handle. With the patch the working set
now consists of fewer mmap windows and mmap-cache is not thrashing.
If the format string contains %m, clearly errno must have a meaningful
value, so we might as well use log_*_errno to have ERRNO= logged.
Using:
find . -name '*.[ch]' | xargs sed -r -i -e \
's/log_(debug|info|notice|warning|error|emergency)\((".*%m.*")/log_\1_errno(errno, \2/'
Plus some whitespace, linewrap, and indent adjustments.
Using:
find . -name '*.[ch]' | while read f; do perl -i.mmm -e \
'local $/;
local $_=<>;
s/(if\s*\([^\n]+\))\s*{\n(\s*)(log_[a-z_]*_errno\(\s*([->a-zA-Z_]+)\s*,[^;]+);\s*return\s+\g4;\s+}/\1\n\2return \3;/msg;
print;'
$f
done
And a couple of manual whitespace fixups.
As a followup to 086891e5c1 "log: add an "error" parameter to all
low-level logging calls and intrdouce log_error_errno() as log calls
that take error numbers", use sed to convert the simple cases to use
the new macros:
find . -name '*.[ch]' | xargs sed -r -i -e \
's/log_(debug|info|notice|warning|error|emergency)\("(.*)%s"(.*), strerror\(-([a-zA-Z_]+)\)\);/log_\1_errno(-\4, "\2%m"\3);/'
Multi-line log_*() invocations are not covered.
And we also should add log_unit_*_errno().
The previous fix e0312f4db "core: fix check for transaction
destructiveness" broke test-engine (noticed by Zbyszek).
Apparently I had a wrong idea of the intended semantics of --fail.
The manpage says the operation should fail if it "conflicts with a
pending job (more specifically: causes an already pending start job to
be reversed into a stop job or vice versa)".
So let's check job_type_is_conflicting, instead of !is_superset.
This makes both test-engine and TEST-03-JOBS pass again.
When checking if the transaction is destructive, we need to check if the
previously installed job is a superset of the new job (and hence the new
job will fold into the installed one without changing it), not the other
way around.
Several functions called from transaction_activate() need to correctly
handle the case where a JOB_NOP job is being checked against a unit's
pending job. The assumption that JOB_NOP never merges with other job
types was correct, but since the job_type_is_*() functions are
implemented using the merge lookup, they need to special-case JOB_NOP
to avoid hitting assertion failures.
Our initrd interface specifies that the verb is in argv[1].
This is where systemd passes it to systemd-shutdown, but getopt
permutes argv[]. This confuses dracut's shutdown script:
Shutdown called with argument '--log-level'. Rebooting!
getopt can be convinced to not permute argv[] by having '-' as the first
character of optstring. Let's use it. This requires changing the way
non-option arguments (in our case, the verb) are processed.
This fixes a bug where the system would reboot instead of powering off.
The SELinux policy defines no context for some files. E.g.:
$ matchpathcon /run/lock/subsys /dev/mqueue
/run/lock/subsys <<none>>
/dev/mqueue <<none>>
We still need to be able to create them.
In this case selabel_lookup_raw() returns ENOENT. We should then skip
setfscreatecon(), but still return success.
It was broken since c34255bdb2 ("label: unify code to make directories,
symlinks").
The option simply enables hashmap debugging by defining
ENABLE_HASHMAP_DEBUG.
I suggest developing new code with it enabled, to have the iterator checks.
This is a rewrite of the hashmap implementation. Its advantage is lower
memory usage.
It uses open addressing (entries are stored in an array, as opposed to
linked lists). Hash collisions are resolved with linear probing and
Robin Hood displacement policy. See the references in hashmap.c.
Some fun empirical findings about hashmap usage in systemd on my laptop:
- 98 % of allocated hashmaps are Sets.
- Sets contain 78 % of all entries, plain Hashmaps 17 %, and
OrderedHashmaps 5 %.
- 60 % of allocated hashmaps contain only 1 entry.
- 90 % of allocated hashmaps contain 5 or fewer entries.
- 75 % of all entries are in hashmaps that use trivial_hash_ops.
Clearly it makes sense to:
- store entries in distinct entry types. Especially for Sets - their
entries are the most numerous and they require the least information
to store an entry.
- have a way to store small numbers of entries directly in the hashmap
structs, and only allocate the usual entry arrays when the direct
storage is full.
The implementation has an optional debugging feature (enabled by
defining the ENABLE_HASHMAP_DEBUG macro), where it:
- tracks all allocated hashmaps in a linked list so that one can
easily find them in gdb,
- tracks which function/line allocated a given hashmap, and
- checks for invalid mixing of hashmap iteration and modification.
Since entries are not allocated one-by-one anymore, mempools are not
used for entries. Originally I meant to drop mempools entirely, but it's
still worth it to use them for the hashmap structs. My testing indicates
that it makes loading of units about 5 % faster (a test with 10000 units
where more than 200000 hashmaps are allocated - pure malloc: 449±4 ms,
mempools: 427±7 ms).
Here are some memory usage numbers, taken on my laptop with a more or
less normal Fedora setup after booting with SELinux disabled (SELinux
increases systemd's memory usage significantly):
systemd (PID 1) Original New Change
dirty memory (from pmap -x 1) [KiB] 2152 1264 -41 %
total heap allocations (from gdb-heap) [KiB] 1623 756 -53 %
It cannot fail in the current hashmap implementation, but it may fail in
alternative implementations (unless a sufficiently large reservation has
been placed beforehand).
With the hashmap implementation that uses chaining the reservations
merely ensure that the merging won't result in long bucket chains.
With a future alternative implementation it will additionally reserve
memory to make sure the merging won't fail.
That hashmap_move_one() currently cannot fail with -ENOMEM is an
implementation detail, which is not possible to guarantee in general.
Hashmap implementations based on anything else than chaining of
individual entries may have to allocate.
hashmap_move_one will not fail with -ENOMEM if a proper reservation has
been made beforehand. Use reservations in install.c.
In cgtop.c simply propagate the error instead of asserting.
With the current hashmap implementation that uses chaining, placing a
reservation can serve two purposes:
- To optimize putting of entries if the number of entries to put is
known. The reservation allocates buckets, so later resizing can be
avoided.
- To avoid having very long bucket chains after using
hashmap_move(_one).
In an alternative hashmap implementation it will serve an additional
purpose:
- To guarantee a subsequent hashmap_move(_one) will not fail with
-ENOMEM (this never happens in the current implementation).
The order of entries may matter here. Oldest entries are evicted first
when the cache is full.
(Though I don't see anything to rejuvenate entries on cache hits.)
-ENOENT is the same return value as if 'other' were an allocated hashmap
that does not contain the key. A NULL hashmap is a possible way of
expressing a hashmap that contains no key.
Test more corner cases and error states in several tests.
Add new tests for:
hashmap_move
hashmap_remove
hashmap_remove2
hashmap_remove_value
hashmap_remove_and_replace
hashmap_get2
hashmap_first
In test_hashmap_many additionally test with an intentionally bad hash
function.
test-hashmap-ordered.c is generated from test-hashmap-plain.c simply by
substituting "ordered_hashmap" for "hashmap" etc.
In the cases where tests rely on the order of entries, a distinction
between plain and ordered hashmaps is made using the ORDERED macro,
which is defined only for test-hashmap-ordered.c.
Few Hashmaps/Sets need to remember the insertion order. Most don't care
about the order when iterating. It would be possible to use more compact
hashmap storage in the latter cases.
Add OrderedHashmap as a distinct type from Hashmap, with functions
prefixed with "ordered_". For now, the functions are nothing more than
inline wrappers for plain Hashmap functions.
Quoting from Jon Corbet's report of Stephen Hemminger's talk at Linux
Plumbers Conference 2014 (https://lwn.net/Articles/616241/):
[...] So Stephen encouraged everybody to run a command like:
sysctl -w net.core.default_qdisc=fq_codel
That will cause fq_codel to be used for all future connections
[Qdiscs apply to interfaces, not connections. Pointed out by TomH
in the article comments. -- mschmidt] (up to the next reboot).
Unfortunately, the default queuing discipline cannot be changed,
since it will certainly disturb some user's workload somewhere.
Let's have the recommended default in systemd.
Thanks to Dave Täht for advice and the summary at
https://lists.bufferbloat.net/pipermail/cerowrt-devel/2014-October/003701.html
Michal Schmidt