The Elastic Stack: Future of ELK Platform

Re:The Elastic Stack: Future of ELK Platform by dozzie on Thursday February 18, 2016 @04:57AM (#11124732)

Good luck with that searching, and remember to share with HN.

I find ElasticSearch somewhat brittle (I need to restart it every few weeks or so, because it stops accepting any data or queries), and I really do want to replace it, since it's a memory hog (no data, and it already needs 230MB RAM), but I haven't found any sensible log storage yet. All I got is this document searching engine.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by Jgrubb on Thursday February 18, 2016 @07:02AM (#11125152)

That was my experience exactly. The value of being able to search through ours logs was immediately apparent, so when it kept tanking every couple weeks it was fairly easy to talk my boss into paying a tiny bit more for Loggly. AFAICT Loggly is the ELK stack with a different theme applied to the UI, and I don't have to worry that it's down when I need it.

The preceding has been an unpaid endorsement for Loggly.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by dozzie on Thursday February 18, 2016 @07:25AM (#11125253)

I would prefer much more to have something hosted by me and being open source at the same time (I'm looking at you, Splunk). I'm desperate enough to write such thing myself (and I'm already doing some research on indexing semi-structured data), even if it would turn out not-too-scalable (I only have ~1.5GB/day of uncompressed logs), but I'm scared by the need of tolerance for abrupt shutdowns and hitting a free disk space limit, i.e., database recoverability.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by Jgrubb on Thursday February 18, 2016 @12:27PM (#11127498)

Hey, I looked at Splunk in the last few days as well, but it seems really expensive once you want to get beyond 500MB a day. They don't have an OSS "community" version, do they (?) because I totally feel you on that.

Basically I need ELK, but I need it to be a lot more stable and documented better.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Thursday February 18, 2016 @10:19AM (#11126328)

Loggly is a five digit monthly bill at our present logging volume. No thanks.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by asafm on Monday February 29, 2016 @10:12AM (#11195866)

I suggest taking a look at the logz.io - it's essentially ELK as a service. We take care of many problems ES has and also have great features extra such as Alerting, Apps and Insights. I'm one of the core developers there so I'm fluent in pricing but last time I checked it's wasn't expensive and definitely worth your time and hosting costs. One the the great things about it is that you can simply migrate your dashboard and just ship to our Url.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by Jgrubb on Thursday February 18, 2016 @12:28PM (#11127510)

What are you using? I'd rather run it myself, but like I said ELK was a bit too bleeding edge for me in the stability dept.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Friday February 19, 2016 @12:17PM (#11134797)

Heka for ingestion/parsing/message routing, RabbitMQ as the queuing/delivery mechanism, Elasticsearch as datastore, Grafana for visualization.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by gibrown on Thursday February 18, 2016 @05:39AM (#11124873)

230MB of Heap is very small for most ES usages, and is likely the reason for you needing to restart it often. I run some very large clusters and even with older versions of ES have seen them run billions of indexing and query operations without problems for 4-6 months without any restarts.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by dozzie on Thursday February 18, 2016 @06:12AM (#11125008)

> 230MB of Heap is very small for most ES usages

230MB RSS, and this is not "ES usage", it's just after starting it with no data whatsoever. I've seen databases which use less memory than that when working.

> I run some very large clusters [...] without problems for 4-6 months without any restarts.

Well, good for you. Being a sysadmin, I understand very well that my ES server may be a specimen combining bad kernel compilation, broken JVM version, unstable ES release, and ground water emanating bad energy, so I don't complain much. Still, I would be very, very happy to see any alternative for storing JSONified logs.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Thursday February 18, 2016 @10:21AM (#11126348)

Despite what they recommend, I'd move to G1GC over CMS. Did wonders for our stability. This is on ES2.1.0 & OpenJDK 1.8.0-u80, IIRC.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by rpedela on Thursday February 18, 2016 @01:17PM (#11127906)

Which version? This reason I ask is that 2.x is a lot more stable for me than previous versions. I have to restart much less often and I haven't had ES "forget" my index since upgrading.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by dozzie on Thursday February 18, 2016 @01:27PM (#11127996)

Some 1.x, and I don't intend to upgrade to 2.x, as it requires Kibana upgrade, and I don't want to deal with 150MB Node.js insanity Kibana 4 brings.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Thursday February 18, 2016 @10:16AM (#11126310)

I'm looking at Graylog right now.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by ktamura on Thursday February 18, 2016 @02:01AM (#11124144)

Curious if you had a chance to look at Fluentd (disclaimer: I'm one of the maintainers of Fluentd) If you did, I'm interested in your honest feedback.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Thursday February 18, 2016 @04:21PM (#11129425)

We already have a transport pipeline; what's needed is a decently scalable, searchable, easy-to-use datastore for short/medium/long-term log analysis.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by packetized on Thursday February 18, 2016 @10:18AM (#11126318)

Not yet, but I'll give it a shot. How's sharding/scalability?

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by zenlikethat on Thursday February 18, 2016 @02:41AM (#11124302)

Right? An ES replacement in Go would be so great.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by sz4kerto on Thursday February 18, 2016 @04:05AM (#11124543)

Why do you care of the language?

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by glial on Thursday February 18, 2016 @07:23AM (#11125238)

For one thing it means you wouldn't have to screw around with the peculiarities of the JVM.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by zenlikethat on Thursday February 18, 2016 @08:23PM (#11130860)

Ridiculously easy deployment and lower memory footprint.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by alexott on Thursday February 18, 2016 @06:47AM (#11125115)

Is there already Lucene for Go? ;-)

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by vruiz on Thursday February 18, 2016 @07:25AM (#11125255)

http://www.blevesearch.com/

It has an very long way ahead to catch up with Lucene, but it's a very promising start.

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by rakoo on Thursday February 18, 2016 @04:04PM (#11129300)

Naïve question from a bystander: how much better is Lucene compared to bleve ?

[ Parent | on HN ]

Re:The Elastic Stack: Future of ELK Platform by eternalban on Thursday February 18, 2016 @12:18PM (#11127388)

/aside: I get rewriting for the n-th time some utility (say a db driver) in language du jour, but (possibly poorly) replicating things like Lucene never made sense. Lucene is a fairly solid tech by all accounts.

[ Parent | on HN ]