This can be streamed and even gzipped in said stream, meaning you get compression and a pretty easy format you can use in most platforms these days with very little intermediate processing or extensions.
With JSON you can have additional metadata on each record, and not have it affect the mainline... now, this cannot be queried directly, but it can be very easily streamed/imported elsewhere.
{"foo": 1}{"bar": 2}
{"foo": 1}
{"bar": 2}
As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case.
Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.
{"dtm":"2016-...",...,meta:{...}}
Maybe as a log archive format, though...
I like SQLite, but I don't think logging is a good use case.
Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?)
I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen.
True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).
And if \n is prohibited in string it is NOT JSON per ecma xyz anymore. it is something else.
Additional complexities that do not seem meaningful but when in congestion they add up to make your life a hell. But well, energy is cheap, VMs are cheap, operational expenses for cloud so less than coders, why bother?
With this kind of reasoning we will have so broken internet appliance coded with feet that we will experience a massive DDOS made by connected toasters and poorly coded cameras.
Like I said, the only place where JSON permits actual newlines is as part of insignificant whitespace. But because it's insignificant, newlines can be replaced with any other whitespace character, and the resulting JSON will have the same exact meaning.
And why would grep lose "order of magnitudes in speed"? If you were actually using grep, it'd work exactly the same as it does for any text file. The only thing that JSON does is add some structure to the contents, so that it can be easily converted to some tabular format that's more convenient for structured queries, aggregation etc. But you can still treat it as plain text for all purposes.
This per ECMA-404.
Crash diagnosis is but one use for logs.
Given the extensive testing including crash testing, it's plausibly more robust than plain text - because you're almost surely not using any kind of fsync's in your plain-text logger, so that text file isn't as incorruptible as you may think. And you may write a buggy logger, or use a buggy json implementation, or write incorrect error-recovery code when reading the file.
I'm skeptical that robustness is an argument in favor of plain text logging over sqlite.
I am, though, and have been for many years.
* http://untroubled.org/daemontools-encore/multilog.8.html
* http://b0llix.net/perp/site.cgi?page=tinylog.8
Agreed. But if the log file is corrupted, then a plain-text one will be easier to decipher for a human than any binary blob.
This is a very important property. It's also what makes UTF-8 resilient.
Netstrings are one option for ephemeral streams. But they are a bad tradeoff for persistent information: you have to read from the beginning of the stream until you find the relevant information. And a single corrupted byte destroys all the information after it.
Better options for persistent data are (pointer+length)s or memory-pools + ranges.
... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
(The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD.
For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX:
http://www.catb.org/esr/writings/taoup/html/.
Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers').
To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
It's not a Good Thing now, though: It's an attack surface.
Also FWIW, I like ESR a lot :) I'm mildly concerned about thread drift, but... why do you dislike him so?
Honestly, he's kind of worse than RMS, who is merely unpleasant to be around...
But racism, and paranoia? I'd (seriously) like to see evidence of those if you have them. Nothing I have seen him write or do suggests he treats people in any way other than as individuals, on their merits alone.
Sounds pretty paranoid to me.
I might have been wrong about the racism stuff. I swear I saw some stuff, but I can't find it now, so I might have been confused with somebody else.
But at the end of the day, he's still an unpleasant person.