I had an automation recipe that would start a bunch of cloud instances, configure them and start certain job via a systemd service unit, and collect logs from this job 24 hours later. Initially, we were pulling these logs via journald as logs for that service unit. But then we noticed that random lines (1-5 lines out of maybe 40 for a single job) would be missing in about half the instances.
We ended up pulling those logs from files in /var/log instead, made courtesy of rsyslog. Not a single missed line.
Again journald is unreliable and as a whole should be nuked from the orbit.
It's not that the logs are missing, it's that they're not getting included by your query.
The way journald associates log lines with process metadata is inherently racy.
This situation has improved some with the metadata caching changes, but IIRC there are still some holes where processes can log without journald associating accurate/up-to-date metadata with the logged info.
The lines would go missing mostly if the logs were collected two or three days later instead of immediately after the job finished (it was a job running for 24 hours), which leads me to think it had to do with journald expiring entries to save on disk space. A consistent log manager would expire them chronologically, but here the lines were missing from the middle or the end of the output, so not the oldest ones logged.
The most common one (and one I share) is how slow it can be. `systemctl status foo.service` takes a significant amount of time on any of my machines thanks to the journal, and that's just the reader side.
It used to be quite crashy with lots of CPU overhead, but that has improved over time, the metadata caching helped a bunch with the overhead.
There was also a pretty embarrassing CVE not long ago:
Many people when they encounter problems with journald want to continue using systemd for their service management, but revert to the old world of low-overhead metadata-free unstructured syslog O_APPEND text files for everything logged.
If systemd (the project) were truly a collection of modular components you could pick and choose, you wouldn't need journald and you'd be able to run syslog like it's the 90s.
It's not an impossibility, the systemd developers just seem to have zero interest in implementing stuff that enables escaping their overall architecture. I imagine all that's needed is some minimal IPC glue to forward all the stdout/stderr output from services to syslog when journald isn't in use. Systemd's role is to wire that stuff up to journald currently, and it's unnecessarily opinionated about journald being the only supported target in that role.
Journald and udevd are exactly the only piece of systemd that systemd's pid 1 requires, and udevd can be used independent of systemd's pid 1.
I just think of journald as a piece of the init system that is split to a separate process, and provides log data to other programs.
Secondly, I don't, I expect the recent logs portion of `systemctl status $unit` to be suppressed when journald isn't present.
Note that journal-caused slowness of that very command is a common complaint aimed at journald contributing to people wanting to remove it!
You can use rsyslog in addition to journald, but there's no way to get journald out of the picture - it's intertwined with the service manager.