News for hackers. Stuff that matters.
Re:Working with the ELK stack by mbell on Friday February 06, 2015 @10:17AM (#9009321)
I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
Re:Working with the ELK stack by robbles on Friday February 06, 2015 @02:05PM (#9010860)
Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria?

We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.

Re:Working with the ELK stack by mbell on Friday February 06, 2015 @07:11PM (#9012417)
Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.
Re:Working with the ELK stack by robbles on Friday February 06, 2015 @07:29PM (#9012496)
Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index?

I assume it would reduce the disk usage as well.

Re:Working with the ELK stack by mbell on Friday February 06, 2015 @09:07PM (#9012778)
Memory usage and CPU time to maintain the index.