Re:Working with the ELK stack
by mbell on Friday February 06, 2015 @10:17AM
(#9009321)
I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
[ Parent
| on HN ]
Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria?
We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.
Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.
Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index?
I assume it would reduce the disk usage as well.