You are underestimating how much costs the issues caused by Kibana and a slow ES can produce.

I used to have these nice Kibana diagrams set up with auth and a https proxy so that our team could easily check if the server error rates looked normal. I quickly found out two things:

1. Most of our employees never ever looked at it, dismissing it as "too technical" just from seeing a screenshot.

2. The ELK stack uses backpressure, meaning the servers running filebeat run out of HDD space.

BTW, with a bit of grep in front, the actual per-service working data sets are small enough so that I can load them into R on a 256GB RAM server and then produce nice diagrams. You can also script R from the command line and then send the diagrams around as a cronjob.

Have you used the file input? With such throughput you don't need it at all, it hangs both of its host because of iowait and remote endpoint too if beats proto is used. All of my log traffic is sent as GELF UDP or syslog UDP.

Sure it’s nice and maybe it’s worth 50k/month if you’re running Uber... but grep and sed together can get you quite far.

Admittedly for the histogram you’d have to cut and paste into a spreadsheet but yeah, hardly something you need in real time (probably more for a postmortem presentation to management.)

You can do this with graylog