Vacuuming behaviour is a bit confusing, and/or we have some bugs,
so those additional messages should help to find out what's going
on. Also, rotation of journal files shouldn't be happening too
often, so the level of the messages is bumped to info, so that
they'll be logged under normal operation.
Make a best-effort attempt to store information about crashes during
failure, currently if these are encountered the crash is completely
silenced.
ideally coredumpctl would show if a coredump is available.
Currently this check happens when the coredump has been collected in
it's entirety and being received by journald. this is not ideal
behaviour when the crashing process is consuming significant percentage
of physical memory such as a large instance of firefox or a java
application.
Before my previous patch, journal_file_empty wasn't be called with the
correct filename. Now that it's being called with the correct filename
it leaks file descriptors. This patch closes the file descriptors before
returning.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
[Edit harald@redhat.com: make use of _cleanup_close_ instead]
d_name is modified on line 227 so if the entire journal name is needed
again p must be used. Before this change when journal_file_empty was called
on archived journals it would always return with -2.
Signed-off-by: George McCollister <george.mccollister@gmail.com>
* Introduce a macro to conditionally execute tests. This avoids
skipping the entire test if some parts require systemd
* Skip the journal tests when no /etc/machine-id is present
* Change test-catalog to load the catalog from the source directory
of systemd.
* /proc/PID/comm got introduced in v2.6.33 but travis is still
using v2.6.32.
* Enable make check and make distcheck on the travis build
* Use -D"CATALOG_DIR=STR($(abs_top_srcdir)/catalog)" as a STRINGIY
would result in the path '/home/ich/source/linux' to be expanded
to '/home/ich/source/1' as linux is defined to 1.
In 49998b383 (journald: do not overwrite syslog facility when
parsing priority) journald started ignoring facility part when
reading service stderr to convert to syslog messages. In this
case it is fine, because only the priority is allowed.
But the same codepath is used for syslog messages, where the
facility should be used. Split the two codepaths by explicitly
specyfing whether the facility should be ignored or not.
https://bugzilla.redhat.com/show_bug.cgi?id=988814
Since the journal can handle multiple lines just well natively,
and rsyslog can be configured to handle them as well, there is no need
to truncate messages from syslog() after the first newline.
Reproducer:
1. Add following four lines to /etc/rsyslog.conf
----------
$EscapeControlCharactersOnReceive off
$ActionFileDefaultTemplate RSYSLOG_SysklogdFileFormat
$SpaceLFOnReceive on
$DropTrailingLFOnReception off
----------
3. Restart rsyslog
# service rsyslog restart
4. Compile and run the following program
----------
#include <stdio.h>
#include <syslog.h>
int main()
{
syslog(LOG_INFO, "aaa%caaa", '\n');
return 0;
}
----------
Actual results:
Below message appears in /var/log/messages.
----------
Sep 7 19:19:39 localhost test2: aaa
----------
Expected results:
Below message, which worked prior to systemd-journald
appears in /var/log/messages.
----------
Sep 7 19:19:39 localhost test2: aaa aaa
https://bugzilla.redhat.com/show_bug.cgi?id=855313
hashmap_free() wasn't being called on m->contexts and m->fds resulting
in a leak.
To reproduce do:
while(1) {
sd_journal_open(&j, SD_JOURNAL_LOCAL_ONLY);
sd_journal_close(j);
}
Memory usage will increase until OOM.
In case of scripts, _EXE is set to the interpreter name, and
_COMM is set based on the file name. Add a match for _COMM,
and _EXE if the interpreter is not a link (e.g. for yum,
the interpreter is /usr/bin/python, but it is a link to
/usr/bin/python2, which in turn is a link to /usr/bin/python2.7,
at least on Fedora, so we end up with _EXE=/usr/bin/python2.7).
I don't think that such link chasing makes sense, because
the final _EXE name is more likely to change.
When using Storage=none there is no point in collecting all the
information just to throw them away. After this change journald
consumes a lot less CPU time when only forwarding messages.
Instead of :-0, :1, :5, etc., use -0, 1 or +1, 5, etc. For BOOT_ID+OFFSET,
use BOOT_ID+offset or BOOT_ID-offset (either + or - is required).
Also make error handling a bit more robust and verbose.
Modify the man page to describe the most common case (-b) first,
and the second most common case (-b -1) second.
A few asserts are replaced with 'return -EINVAL'. I think that
assert should not be used to check argument in public functions.
Fields in struct sd_journal are rearranged to make it less
swiss-cheesy.
Hi,
I redid the boot ID look up to use enumerate_unique.
This is quite fast if the cache is warm but painfully slow if
it isn't. It has a slight chance of returning the wrong order if
realtime clock jumps around.
This one has to do n searches for every boot ID there is plus
a sort, so it depends heavily on cache hotness. This is in contrast
to the other way of look-up through filtering by a MESSAGE_ID,
which only needs about 1 seek + whatever amount of relative IDs
you want to walk.
I also have a linked-list + (in-place) mergesort version of this
patch, which has pretty much the same runtime. But since this one
is using libc sorting and armortized allocation, I prefer this
one.
To summarize: The MESSAGE_ID way is a *lot* faster but can be
incomplete due to rotation, while the enumerate+sort will find
every boot ID out there but will be painfully slow for large
journals and cold caches.
You choose :P
Jan
Generating seed...
Generating key pair...
Generating sealing key...
Failed to open /var/log/journal/33f46101703a10c5fc6fa4f451840101/fss.tmp.k2wDDU: No such file or directory
Corrupted empty files are relatively common. I think they are created
when a coredump for a user who never logged anything before is
attempted to be written, but the write does not succeed because the
coredump is too big, but there are probably other ways to create
those, especially if the machine crashes at the right time.
Non-corrupted empty files can also happen, e.g. if a journal file is
opened, but nothing is ever successfully written to it and it is
rotated because of MaxFileSec=. Either way, each "empty" journal file
costs around 3 MB, and there's little point in keeping them around.
Reporting of the free space was bogus, since the remaining space
was compared with the maximum allowed, instead of the current
use being compared with the maximum allowed. Simplify and fix
by reporting limits directly at the point where they are calculated.
Also, assign a UUID to the message.
Sometimes an entry is not successfully written, and we end up with
data items which are "unlinked", not connected to, and not used by any
entry. This will usually happen when we write to write a core dump,
and the initial small data fields are written successfully, but
the huge COREDUMP= field is not written. This situation is hard
to avoid, but the results are mostly harmless. Thus only warn about
unused data items.
Also, be more verbose about why journal files failed verification.
This should help diagnose journal failure modes without resorting
to a hexadecimal editor.
https://bugs.freedesktop.org/show_bug.cgi?id=65235 (esp. see
system.journal attached to the bug report).
When journald encounters a message with OBJECT_PID= set
coming from a priviledged process (UID==0), additional fields
will be added to the message:
OBJECT_UID=,
OBJECT_GID=,
OBJECT_COMM=,
OBJECT_EXE=,
OBJECT_CMDLINE=,
OBJECT_AUDIT_SESSION=,
OBJECT_AUDIT_LOGINUID=,
OBJECT_SYSTEMD_CGROUP=,
OBJECT_SYSTEMD_SESSION=,
OBJECT_SYSTEMD_OWNER_UID=,
OBJECT_SYSTEMD_UNIT= or OBJECT_SYSTEMD_USER_UNIT=.
This is for other logging daemons, like setroubleshoot, to be able to
augment their logs with data about the process.
https://bugzilla.redhat.com/show_bug.cgi?id=951627
Since the system journal wasn't open yet, available_space() returned 0.
Before:
systemd-journal[22170]: Allowing system journal files to grow to 4.0G.
systemd-journal[22170]: Journal size currently limited to 0B due to SystemKeepFree.
After:
systemd-journal[22178]: Allowing system journal files to grow to 4.0G.
systemd-journal[22178]: Journal size currently limited to 3.0G due to SystemKeepFree.
Also, when failing to write a message, show how much space was needed:
"Failed to write entry (26 items, 260123456 bytes) despite vacuuming, ignoring: ...".
In the following scenario:
server creates system.journal
server creates user-1000.journal
both journals share the same seqnum_id.
Then
server writes to user-1000.journal first,
and server writes to system.journal a bit later,
and everything is fine.
The server then terminates (crash, reboot, rsyslog testing,
whatever), and user-1000.journal has entries which end with
a lower seqnum than system.journal. Now
server is restarted
server opens user-1000.journal and writes entries to it...
BAM! duplicate seqnums for the same seqnum_id.
Now, we usually don't see that happen, because system.journal
is closed last, and opened first. Since usually at least one
message is written during boot and lands in the system.journal,
the seqnum is initialized from it, and is set to a number higher
than than anything found in user journals. Nevertheless, if
system.journal is corrupted and is rotated, it can happen that
an entry is written to the user journal with a seqnum that is
a duplicate with an entry found in the corrupted system.journal~.
When browsing the journal, journalctl can fall into a loop
where it tries to follow the seqnums, and tries to go the
next location by seqnum, and is transported back in time to
to the older duplicate seqnum. There is not way to find
out the maximum seqnum used in a multiple files, without
actually looking at all of them. But we don't want to do
that because it would be slow, and actually it isn't really
possible, because a file might e.g. be temporarily unaccessible.
Fix the problem by using different seqnum series for user
journals. Using the same seqnum series for rotated journals
is still fine, because we know that nothing will write
to the rotated journal anymore.
Likely related:
https://bugs.freedesktop.org/show_bug.cgi?id=64566https://bugs.freedesktop.org/show_bug.cgi?id=59856https://bugs.freedesktop.org/show_bug.cgi?id=64296https://bugs.archlinux.org/task/35581https://bugzilla.novell.com/show_bug.cgi?id=817778
Possibly related:
https://bugs.freedesktop.org/show_bug.cgi?id=64293
