journal: enforce strict consistency for realtime timestamps on write

Ensure all realtime timestamps in a journal file are strictly
ordered on write as a defense-in-depth measure. All known callers of
journal_file_append_entry and journal_file_copy_entry, which call this
function, should be able to handle the error by rotating the journal.

This is especially helpful for systems with RTC local time enabled,
where all log entries from initramfs might be recorded as several hours
later than it actually is, which won't get caught by journald during log
flushing. In those cases, the resulting inconsistency can cause libsystemd
to loop infinitely through journal files as observed in
`abrt-dump-journal-oops`.

src/journal/journald-server.c

@@ -839,6 +839,10 @@ static bool shall_try_append_again(JournalFile *f, int r) {
                 log_ratelimit_warning(JOURNAL_LOG_RATELIMIT, "%s: Journal file is from the future, rotating.", f->path);
                 return true;
 
+        case -EREMCHG:         /* Time jumped backwards relative to last journal entry */
+                log_ratelimit_warning(JOURNAL_LOG_RATELIMIT, "%s: Time jumped backwards relative to last journal entry, rotating.", f->path);
+                return true;
+
         case -EAFNOSUPPORT:
                 log_ratelimit_warning(JOURNAL_LOG_RATELIMIT,
                                       "%s: underlying file system does not support memory mapping or another required file system feature.",

src/libsystemd/sd-journal/journal-file.c

@@ -2092,6 +2092,12 @@ static int journal_file_append_entry_internal(
         assert(ts);
         assert(items || n_items == 0);
 
+        if (ts->realtime < le64toh(f->header->tail_entry_realtime))
+                return log_debug_errno(SYNTHETIC_ERRNO(EREMCHG),
+                                       "Realtime timestamp %" PRIu64 " smaller than previous realtime "
+                                       "timestamp %" PRIu64 ", refusing entry.",
+                                       ts->realtime, le64toh(f->header->tail_entry_realtime));
+
         osize = offsetof(Object, entry.items) + (n_items * journal_file_entry_item_size(f));
 
         r = journal_file_append_object(f, OBJECT_ENTRY, osize, &o, &np);