I would prefer much more to have something hosted by me and being open source at the same time (I'm looking at you, Splunk). I'm desperate enough to write such thing myself (and I'm already doing some research on indexing semi-structured data), even if it would turn out not-too-scalable (I only have ~1.5GB/day of uncompressed logs), but I'm scared by the need of tolerance for abrupt shutdowns and hitting a free disk space limit, i.e., database recoverability.

Hey, I looked at Splunk in the last few days as well, but it seems really expensive once you want to get beyond 500MB a day. They don't have an OSS "community" version, do they (?) because I totally feel you on that.

Basically I need ELK, but I need it to be a lot more stable and documented better.

Loggly is a five digit monthly bill at our present logging volume. No thanks.

I suggest taking a look at the logz.io - it's essentially ELK as a service. We take care of many problems ES has and also have great features extra such as Alerting, Apps and Insights. I'm one of the core developers there so I'm fluent in pricing but last time I checked it's wasn't expensive and definitely worth your time and hosting costs. One the the great things about it is that you can simply migrate your dashboard and just ship to our Url.

What are you using? I'd rather run it myself, but like I said ELK was a bit too bleeding edge for me in the stability dept.

Heka for ingestion/parsing/message routing, RabbitMQ as the queuing/delivery mechanism, Elasticsearch as datastore, Grafana for visualization.