News for hackers. Stuff that matters.
Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @06:28PM (#2826981)
It's queryable.

It's trivially capped and rolled.

It's centralized.

It provides a common logging interface across platforms.

It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long).

I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate.

Have you noticed how none of the cool kids postprocess http log files anymore?

Re:Another Redis case: Centralized logging by cdavid on Saturday July 30, 2011 @08:08PM (#2827165)
I don't really understand your list in view of the article. How is this solution more queryable than syslog: they record events into redis without any schema related to it (just a string), so I fail to see the improvement there. They put it back to a log file anyway.

It is not less or more centralized than syslog configured with centralization (which is trivial to set up).

How is this more common than syslog across platforms (unless you include windows in "across platforms ?").

It is not simpler than syslog either, since writing to syslog is just a matter of using the right python logging backend.

Analysing tons of data from syslog is a pain, but I don't see how any solution will not require at some point in the stack to enforce a format/structure in your log. How is this fundamentally different than post-processing http log ?

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:13PM (#2827175)
He's LPUSH'ing logs into a list key. Just by doing that one thing, he now has evented logging; he can subscribe to his raw logs with BRPOPLPUSH and without changing anything clientside start indexing them in better ways, or applying different policies to different logs.

And Redis is easier to understand than syslog. We're pretending that there is zero friction to understanding syslog, as if any competent Unix person should automatically grok it. But syslog is a janky old rube goldberg machine and understanding it well pays off solely in the form of understanding a janky old rube goldberg machine that nobody will be using in 10 years.

Re:Another Redis case: Centralized logging by mihasya on Saturday July 30, 2011 @09:33PM (#2827328)
The lady doth protest too much, no? The guy is using Redis as a buffer to what ultimately ends up just being a central log file. You know. rsyslog. None of the fancy things you have in mind. Which brings me to the same point someone's already made: using Redis buys you nothing, even if you had a more involved use case in mind.

Saying that he can do something by subscribing to an event in Redis is sorta silly, isn't it? You could just as easily tail the logs, as many systems actually built for this purpose do. Once again, there are already things in *nix for this.

The reality is, this is a janky, nubile solution to an already well-solved problem that is now getting thousands of views because of HN and antirez tweeting about it. Instead of learning the correct way of doing things, I bet a bunch of inexperienced developers are now going to say "BOY THIS IS SWELL" and cut themselves a nice fat chunk of technical debt for the not-that-far-off future.

Re:Another Redis case: Centralized logging by cdavid on Saturday July 30, 2011 @08:47PM (#2827239)
I don't understand how BRPOPLPUSH helps you doing things you could not do with centralized syslog: you can also start indexing things with syslog by analysing the central logs without changing anything client-side. And in both cases, what will be needed is essentially the same: if the logs are not structured at the source, you will need to post-process them, the storage medium does not change any of that.

The redis is easier than syslog is a bit of a strawman, because you will have to understand syslog anyway, since that's the only thing spoken by quite a few applications. In the OP'case, they are already using redis, so the cost on that side is very low, though.

Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:07PM (#2827160)
I think you're missing the forest for the trees.

What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:10PM (#2827168)
Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:29PM (#2827203)
Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel?

Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:30PM (#2827204)
If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:31PM (#2827206)
No special new logging protocol is required.

So, what do I put into the redis configuration-file to make it log to itself?

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:36PM (#2827213)
Is the suggestion here that because Redis logs suboptimally, Redis is a bad solution for managing log entries?
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:52PM (#2827249)
No, the suggestion is: Having all logs in one place is a good thing. Even if it's so unstructured that only a human can digest it.

Context matters, a lot. When my app logs a timeout against redis then my next question is "so what did redis do at the time, did it perhaps log something"?

Following your advice I'd either have to look in two places (redislog and syslog) or feed my syslog stream into redislog, to have everything end up in one place (redis).

Any sane person would do the latter. Under that premise, what's the benefit of having some apps log to redis directly when the syslog-stream also ends up in redis anyways?