News for hackers. Stuff that matters.
Re:Debian reconsiders init-system diversity by pferde on Tuesday November 12, 2019 @01:59AM (#21512093)
My main complaint is that journald is unreliable and is losing log entries (or lines, if you will).

I had an automation recipe that would start a bunch of cloud instances, configure them and start certain job via a systemd service unit, and collect logs from this job 24 hours later. Initially, we were pulling these logs via journald as logs for that service unit. But then we noticed that random lines (1-5 lines out of maybe 40 for a single job) would be missing in about half the instances.

We ended up pulling those logs from files in /var/log instead, made courtesy of rsyslog. Not a single missed line.

Again journald is unreliable and as a whole should be nuked from the orbit.

Re:Debian reconsiders init-system diversity by Yizahi on Tuesday November 12, 2019 @07:37AM (#21513557)
I concur. We also saw an issue with journald missing whole hours of logs, randomly, on our embedded platform. Couldn't replace all of them with log files due to SSD degradation (we actually calculated it we were missing target device lifespan, this is not a consumer problem of course). Basically we added log files where we could, plus remote logs and left journald as-is, it can't be replaced and can't be fixed.
Re:Debian reconsiders init-system diversity by newnewpdro on Tuesday November 12, 2019 @02:22AM (#21512183)
Filtering journal contents by metadata is known to be unreliable.

It's not that the logs are missing, it's that they're not getting included by your query.

The way journald associates log lines with process metadata is inherently racy.

This situation has improved some with the metadata caching changes, but IIRC there are still some holes where processes can log without journald associating accurate/up-to-date metadata with the logged info.

Re:Debian reconsiders init-system diversity by pferde on Tuesday November 12, 2019 @03:12AM (#21512424)
That makes sense, I guess, in a rather warped and bizarre way. The point still stands - journald is unreliable, and after getting burnt like this it will be years before I start trusting it again.
Re:Debian reconsiders init-system diversity by newnewpdro on Tuesday November 12, 2019 @12:30PM (#21515988)
FYI, in case you wanted to document your pain, it might help prioritize fixing it fully:

https://github.com/systemd/systemd/issues/2913

Re:Debian reconsiders init-system diversity by dannyw on Tuesday November 12, 2019 @05:27AM (#21513029)
Great, so why don't we just keep text logs as text files?
Re:Debian reconsiders init-system diversity by mateuszf on Tuesday November 12, 2019 @06:08AM (#21513209)
Filtering text files in a structured way is even harder.
Re:Debian reconsiders init-system diversity by pmoriarty on Tuesday November 12, 2019 @10:18AM (#21514726)
Having unstructured text reliably be in log files is far better than having a broken log daemon forced upon you.
Re:Debian reconsiders init-system diversity by eecc on Wednesday November 20, 2019 @03:10AM (#21582017)
Dedicated tools exist for that...
Re:Debian reconsiders init-system diversity by freedomben on Tuesday November 12, 2019 @02:02AM (#21512104)
Interesting, this is the first time I've ever heard this. How long ago did this happen?
Re:Debian reconsiders init-system diversity by pferde on Tuesday November 12, 2019 @02:11AM (#21512131)
Sometimes between August and September, and it was on Suse Linux Enterprise Server 12.

The lines would go missing mostly if the logs were collected two or three days later instead of immediately after the job finished (it was a job running for 24 hours), which leads me to think it had to do with journald expiring entries to save on disk space. A consistent log manager would expire them chronologically, but here the lines were missing from the middle or the end of the output, so not the oldest ones logged.