I believe Loki [1] is intended to basically run "grep foo" at scale (plus some extra niceties like labels). I haven't used it, but it seems interesting.
ELK stack user here - we actually found logstash to be our bottleneck. Changing it out for fluentd fixed our woes.
Same here, fluentd is much better, performance wise.
But then I had to give ES more RAM because it couldn't take the hammering.
In fact, increasing the throughput to ES was causing some pretty spectacular crashes, with the /var/log partition at 100% because of the verbosity of the dumps.
Re:Security for Elasticsearch is now free
by majkinetor on Tuesday May 21, 2019 @05:51AM
(#19968762)
Logstash sucks from both operational and developing perspective. I replaced it everywhere I could by sending
structured logs directly from the app or by using newer integrated beats features.
The conclusion that I've reached is that the whole lucene model for logs is kind of outdated. Why am I tuning Java GC params to run "grep foo /logs". I think computers today can do fine with sharded flat files, a minimal index ("which node contains logs from pod foo-2387438-2384738 at 12:34AM"), and then just scale horizontally over (log messages, searches).
I hope my friends over at Tailscale are doing that and I can just move off ES entirely ;)