Re:Command-line tools can be faster than a Hadoop cluster (2...
by fxtentacle on Thursday January 30, 2020 @02:27AM
(#22189584)
We used a central logstash server to receive the data from filebeat and logstash-forwarder running on multiple other servers. So in the initial deployment, logstash only set up the host and service tags needed for Kibana plus it was like a central gateway for external servers to write into the locally running ES storage.
As for why the price that elastic.co quoted us was lower than EC2, I have no idea. My guess would be that they were hoping to get a reference customer onboard for their (then new) offering.
Or maybe they are internally running bare-metal and thereby purchasing their CPU power cheaper than EC2 pricing. I mean you don't really need a redundant fault-tolerant virtual machine if the database on top is redundant and fault-tolerant, too.
And secondly - why use logstash if you had filebeat - why not send data to ES directly? (I mean, specifically for your usecase, where you don't seem to need to do much processing prior to ingestion). Yeah, I saw that big parts of Logstash are in ruby so I expect underwhelming performance from it - but Filebeat should in theory be able to ingest large volumes of data into ES directly.