News for hackers. Stuff that matters.
Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @06:22PM (#2826968)
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too.

* Freeform text is a terrible way to track system events.

* Periodically rotated flat files are not a great way to store log information.

* Goofy little UDP messages are not a good way to convey system events

* The syslog PRI field dates back to when we exchanged messages with UUCP.

I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.

Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @07:57PM (#2827148)
Yes, syslog is a pile of shit. It's a relic.

And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running.

However...

Freeform text is a terrible way to track system events.

Nothing stops you from logging structured text.

Periodically rotated flat files are not a great way to store log information.

Modern syslog daemons will write to pretty much anything you want.

Goofy little UDP messages are not a good way to convey system events

Modern syslog daemons offer tcp transport. Some even try to offer some delivery guarantees (disk-backed spool), although personally I wouldn't rely on that for truly critical stuff.

The syslog PRI field dates back to when we exchanged messages with UUCP.

Thanks, I always wondered where those were from...

And, well, you forgot a couple bullets:

* syslog() is available everywhere, out of the box

* It's trivial to move from file-based logging to syslog

* We have mature syslog-daemons that dispatch events pretty reliably

* Unless you're facebook you probably don't need anything more fancy.

So, I'd say syslog gets the job done quite well, as long as you don't mistake it for a message queue.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:05PM (#2827154)
1.

It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on.

2.

Most syslog daemons will store anywhere. Sure. The last syslog daemon I actually read had stack overflows in it, so it's been awhile for me. But if you've got syslog writing to a real store, what value is syslog adding? The standardized network protocol? Redis has a trivial network protocol too.

3.

The issue with syslog facilities isn't that the "UUCP" facility harms anything directly; it's that one the few bits of out-of-band metadata syslog truly offers forces applications to decide whether they're "kern" or "daemon" or "local2". Whose application actually breaks down cleanly into syslog facilities?

4.

I dispute that syslog is "good enough if you're not Facebook". I think you do too: you probably aren't doing all your metrics stuff with syslog logs; you probably aren't even getting your web stats though syslog. You may have even delegated your web logs to client side Javascript, because that is how bad text logs stored in flat files are to work with.

Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:25PM (#2827199)
what value is syslog adding? The standardized network protocol?

Exactly. Every piece of software under the sun either logs to syslog by default or has a config-switch/patch to make it so.

Yes, syslog the protocol is pretty nasty. Yes, syslog daemons are pretty nasty. But they get the job done and you are free to put anything (including redis) at the end of the pipe.

Coping with syslog is by far easier than trying to make all "legacy software" including the kernel speak something else entirely.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:27PM (#2827201)
But who's suggesting that? Nobody is saying "let's have the kernel log to Redis". By all means, hot potato the kernel stuff and the wrapper stuff and your authlog and whatever through syslog before it gets dumped into Redis.

But if you control it, why bother with syslog? Syslog is a piece of junk. Don't bother. Any ad hoc scheme you come up with that uses a real backend store will be better than syslog.

Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:45PM (#2827234)
But if you control it, why bother with syslog?

I'd ask this backwards: Why bother with a homegrown solution when syslog exists and gets the job done?

A home-grown solution takes at least a couple weeks to stabilize, likely much longer before the last bugs are ironed out. Syslog takes about a day to beat into shape.

Any ad hoc scheme you come up with that uses a real backend store will be better than syslog.

You know better than that.

Shipping messages reliably is a surprisingly tedious problem. First you realize you need a disk-spool. Then you realize that spool should be size-capped anyhow. Eventually your boss says you need a network topology more complex than A->B for some idiotic but inevitable reason.

And then next month you run into some redis limitation and realize some kind of datastore abstraction would have been a better idea to begin with. Hmm, perhaps dump to plain-text files until we figure that one out?

See the pattern here?

At the end you have reinvented syslog. Sure, yours may be nicer or at least different.

But that's a whole lot of work to avoid something that, despite all its warts, already works.

Re:Another Redis case: Centralized logging by Andys on Saturday July 30, 2011 @07:02PM (#2827053)
There's syslog the protocol, syslog the API, and syslog the daemon. There are syslog daemon and protocol replacements that are much better, but retain the API for wide-spread compatibility.
Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @07:17PM (#2827076)
I think my contention would be that if you shake off the early-90s-Unixisms and adopt any modern store (forget Redis even and just "use sqlite instead of fopen"), the machinery required to do 98% of what the best syslog daemons do is trivial. The syslog infrastructure just doesn't add much... except compatibility with a very dubious ecosystem of tools.
Re:Another Redis case: Centralized logging by zbailey on Saturday July 30, 2011 @06:32PM (#2826990)
We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog.

Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @07:19PM (#2827079)
I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.