News for hackers. Stuff that matters.
Re:Working with the ELK stack by nissimk on Friday February 06, 2015 @09:33AM (#9009113)
Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you are just talking about elasticsearch. Is there a similar solution that works well in just a couple of gigs of RAM for a slower message flow rate?

Is anybody here successfully using ELK in a low ram environment with low message flow?

Re:Working with the ELK stack by snewman on Friday February 06, 2015 @11:02AM (#9009567)
If you're open to commercial solutions, there are several SaaS choices that are quite cheap at low log volume. For instance, Scalyr (500MB/day for $19/month), LogEntries (free up to 167MB/day), or Papertrail (33MB/day for $7/month). All will provide centralized logging and full text search, and some form of dashboards and alerting.

Scalyr provides especially powerful features for log parsing and analysis, as well as integrating system and application metrics, and it's wicked fast -- most searches run in well under 1 second. (Disclosure: I am the founder of Scalyr.)

If you're interested, the respective web sites are easy to find, or drop me a line (my email address is in my profile).

Re:Working with the ELK stack by 23david on Friday February 06, 2015 @03:07PM (#9011214)
For most teams, I definitely recommend getting started with a hosted elasticsearch option initially, and after validating a POC, only then consider a move to self-managed. Check out https://bonsai.io as a hosted option...

I've helped a bunch of financial and security companies implement and deploy elk as a critical infra component. Generally I'm seeing it used to complement Splunk, Alienvault and other log/network/security monitoring solutions.

Happy to help with any questions about deploying and managing ELK in production.

Re:Working with the ELK stack by mbell on Friday February 06, 2015 @10:17AM (#9009321)
I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
Re:Working with the ELK stack by robbles on Friday February 06, 2015 @02:05PM (#9010860)
Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria?

We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.

Re:Working with the ELK stack by mbell on Friday February 06, 2015 @07:11PM (#9012417)
Nothing special on the ES config. Indexes are split up by day and we avoid indexing fields with very high cardinality.
Re:Working with the ELK stack by robbles on Friday February 06, 2015 @07:29PM (#9012496)
Thanks for the info. Does avoiding indexing high cardinality fields keep down the memory usage of an open index?

I assume it would reduce the disk usage as well.

Re:Working with the ELK stack by mbell on Friday February 06, 2015 @09:07PM (#9012778)
Memory usage and CPU time to maintain the index.
Re:Working with the ELK stack by ihsw on Friday February 06, 2015 @10:13AM (#9009300)
Please define low RAM.

I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load.

Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors.

So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.

Re:Working with the ELK stack by Iaks on Friday February 06, 2015 @12:52PM (#9010363)
I am running a monolithic installation in a container (1 core, 4gb ram) that is being hosted by an intel NUC. I'm taking in about 100k records a day from 10-12 syslog sources and have no trouble with slow downs at all. (I do almost 0 processing of the incoming data, however.) I think I could lower the containers ram allocation to 2gb... maybe.

I haven't tried aggregating a years worth of data into a single search, but for reasonable log troubleshooting everything is going well.

Re:Working with the ELK stack by simonw on Friday February 06, 2015 @10:02AM (#9009254)
Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.
Re:Working with the ELK stack by kiyoto on Friday February 06, 2015 @12:58PM (#9010401)
Some people like using Grafana with InfluxDB backend and Fluentd for log collection.