My team is using Loki + Grafana and we're pretty happy with it. It's pretty basic but it does what you expect it to do just fine.

We ditched Elastic as it was a super massive PITA to operate (& a resource hog at that). I'll admit I'm not an expert at ELK at all, but tbh I was absolutely surprised just how bad Elastic + Kibana was for our basic log uses when they tout it as one of their mainstays. Or we were just exceptionally stupid, who knows. In any case, the experience we had with it didn't motivate us to become ELK experts at all. Our pet peeves:

- The Kibana UI needlessly wastes tons screen for whitespace

- makes it hard to dig down into logs

- never seems to find exact string matches when we wanted it to and instead returns "helpful" fuzzy matches

- Kibana has no qualms sending requests to Elastic that will happily kill your node instead of applying sensible paging / query timeouts. I mean that's why I'm using Kibana and not writing my own elastic frontend...

How has your experience with Loki been? I keep an eye on it, but haven't deployed it since it seems to be maturing still.

Running Loki with multiple instances is insanely hard

I mean, that product is called LogDNA.

But Elasticsearch has evolved into a whole bunch of things to meet everyone's needs. There's a way to do what you want and simply, but you have to find the simple path in the middle of the big product.

I mean, the same reason that SQL isn't any simpler? It takes arbitrary data, and you can filter and aggregate it in arbitrary ways? People aren't just doing log data... it's just been heavily adopted for that purpose

I think the "simpler" version of ES+Kibana is probably a spreadsheet.

I built https://logflare.app for exactly this.

Solr is quite a bit more straightforward than ElasticSearch, but still a bit complicated in my opinion.

ElasticSearch is really just clustered Lucene with some nice features wrapping it. You can probably get away with something that also wraps lucene. Though elasticsearch has a dominant position precisely because it is quite full featured and easy to run.

Im afraid its a bit over simplification . For example - Aggregations in elasticsearch are not using lucene facets, but it just leverages the basic lucene mechanisms ( collector) .

I think the question you should be asking is, does my use case require a tool as powerful as ES?

And only you can answer that fully.

> JSON logs with a timestamp + a bunch of fields then search them

There is S3+Athena for this with AWS and Google can store/query JSON with BigQuery. The nice little UI doesn't come with it, but at least you don't have to spin up an Elastic cluster.

Do you have a S3 + Athena example because when I tried it didn't seem to query adhock json file but rather the S3 file needs contain a array of json documents.

aws cloudwatch log insights can do a lot natively without the s3/athena step now

Splunk is that product, but it’s only free up to 500MB/day of indexing

if the logs are only a few GB you can still write to files and fgrep.