Command-line tools can be faster than a Hadoop cluster (2014)

Re:Command-line tools can be faster than a Hadoop cluster (2... by Spivak on Thursday January 30, 2020 @10:19AM (#22191885)

rsyslog is far more powerful than you're making it out. You have to actually tell it what to do but it's more expressive than filebeat and logstash.

* rsyslog in the use-case he's describing is just a method of pushing some subset of the logs generated on a client system to a directory on the collector which has trade-offs but the benefit is having really simple failure modes.

* both rsyslog and journald store structured data: rsyslog with lumberjack, and journald just always. And rsyslog can parse and structure the logs in-flight so you save computing power on the collector.

* rsyslog behaves exactly like filebeat when it comes to reliable delivery and can persist unsent messages to memory then disk. rsyslog's rate limiting, backoff, and retry options are super powerful.

[ Parent | on HN ]

Re:Command-line tools can be faster than a Hadoop cluster (2... by pepemon on Thursday January 30, 2020 @10:54AM (#22192242)

Yes, you are right, sorry. I was too fast in my assumptions, rsyslog (don't know much of syslog-ng) has feature parity with ELK in terms of log delivery and processing. But I think that grep and its permutations aren't right tools of choice for log analysis anyway.

[ Parent | on HN ]