News for hackers. Stuff that matters.
Re:Another Redis case: Centralized logging by andrewvc on Saturday July 30, 2011 @04:16PM (#2826626)
My only question: how is this better than syslog?
Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @06:28PM (#2826981)
It's queryable.

It's trivially capped and rolled.

It's centralized.

It provides a common logging interface across platforms.

It's extremely simple (the script that pipes syslog output directly into Redis is probably just a couple lines long).

I have to believe that the people who really seem to like syslog have never worked in organizations that had to deploy things like Splunk or (worse) LogLogic and ArcSight just to make sense of the giant morass of useless text gunk they generate.

Have you noticed how none of the cool kids postprocess http log files anymore?

Re:Another Redis case: Centralized logging by cdavid on Saturday July 30, 2011 @08:08PM (#2827165)
I don't really understand your list in view of the article. How is this solution more queryable than syslog: they record events into redis without any schema related to it (just a string), so I fail to see the improvement there. They put it back to a log file anyway.

It is not less or more centralized than syslog configured with centralization (which is trivial to set up).

How is this more common than syslog across platforms (unless you include windows in "across platforms ?").

It is not simpler than syslog either, since writing to syslog is just a matter of using the right python logging backend.

Analysing tons of data from syslog is a pain, but I don't see how any solution will not require at some point in the stack to enforce a format/structure in your log. How is this fundamentally different than post-processing http log ?

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:13PM (#2827175)
He's LPUSH'ing logs into a list key. Just by doing that one thing, he now has evented logging; he can subscribe to his raw logs with BRPOPLPUSH and without changing anything clientside start indexing them in better ways, or applying different policies to different logs.

And Redis is easier to understand than syslog. We're pretending that there is zero friction to understanding syslog, as if any competent Unix person should automatically grok it. But syslog is a janky old rube goldberg machine and understanding it well pays off solely in the form of understanding a janky old rube goldberg machine that nobody will be using in 10 years.

Re:Another Redis case: Centralized logging by mihasya on Saturday July 30, 2011 @09:33PM (#2827328)
The lady doth protest too much, no? The guy is using Redis as a buffer to what ultimately ends up just being a central log file. You know. rsyslog. None of the fancy things you have in mind. Which brings me to the same point someone's already made: using Redis buys you nothing, even if you had a more involved use case in mind.

Saying that he can do something by subscribing to an event in Redis is sorta silly, isn't it? You could just as easily tail the logs, as many systems actually built for this purpose do. Once again, there are already things in *nix for this.

The reality is, this is a janky, nubile solution to an already well-solved problem that is now getting thousands of views because of HN and antirez tweeting about it. Instead of learning the correct way of doing things, I bet a bunch of inexperienced developers are now going to say "BOY THIS IS SWELL" and cut themselves a nice fat chunk of technical debt for the not-that-far-off future.

Re:Another Redis case: Centralized logging by cdavid on Saturday July 30, 2011 @08:47PM (#2827239)
I don't understand how BRPOPLPUSH helps you doing things you could not do with centralized syslog: you can also start indexing things with syslog by analysing the central logs without changing anything client-side. And in both cases, what will be needed is essentially the same: if the logs are not structured at the source, you will need to post-process them, the storage medium does not change any of that.

The redis is easier than syslog is a bit of a strawman, because you will have to understand syslog anyway, since that's the only thing spoken by quite a few applications. In the OP'case, they are already using redis, so the cost on that side is very low, though.

Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:07PM (#2827160)
I think you're missing the forest for the trees.

What you really want to do (and what everyone does btw) is to push your logs to a central syslog-server and stream them into redis or whatever analytics solution from there.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:10PM (#2827168)
Why am I pushing logs through the crufty old syslog system instead of just directly streaming them into Redis? Redis is really good at this. Syslog isn't really good at anything.
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:29PM (#2827203)
Counter-question: Why am I running my sexy sweet logging solution and syslog in parallel?

Because that's exactly what I'd have to do until other daemons like, say, redis itself, speak this fancy new protocol.

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:30PM (#2827204)
If there was a fancy new protocol actually involved, I'd agree with you, but this is a case of simply using an existing tool for a problem it is very well suited to. No special new logging protocol is required.
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:31PM (#2827206)
No special new logging protocol is required.

So, what do I put into the redis configuration-file to make it log to itself?

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @08:36PM (#2827213)
Is the suggestion here that because Redis logs suboptimally, Redis is a bad solution for managing log entries?
Re:Another Redis case: Centralized logging by moe on Saturday July 30, 2011 @08:52PM (#2827249)
No, the suggestion is: Having all logs in one place is a good thing. Even if it's so unstructured that only a human can digest it.

Context matters, a lot. When my app logs a timeout against redis then my next question is "so what did redis do at the time, did it perhaps log something"?

Following your advice I'd either have to look in two places (redislog and syslog) or feed my syslog stream into redislog, to have everything end up in one place (redis).

Any sane person would do the latter. Under that premise, what's the benefit of having some apps log to redis directly when the syslog-stream also ends up in redis anyways?

Re:Another Redis case: Centralized logging by theonlyroot on Saturday July 30, 2011 @04:29PM (#2826665)
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
Re:Another Redis case: Centralized logging by trentonstrong on Saturday July 30, 2011 @05:49PM (#2826879)
Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways.

I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible.

On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!

Re:Another Redis case: Centralized logging by tptacek on Saturday July 30, 2011 @07:28PM (#2827094)
Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs.

The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.

Re:Another Redis case: Centralized logging by tedjdziuba on Saturday July 30, 2011 @07:05PM (#2827059)
Ahem.

tail -n 1000