News for hackers. Stuff that matters.
Re:A log/event processing pipeline you can't have (2019) by d4rti on Wednesday August 26, 2020 @06:12AM (#24281008)
elasticsearch would not cope with that volume of data on the hardware described.
Re:A log/event processing pipeline you can't have (2019) by kakoni on Wednesday August 26, 2020 @09:22AM (#24282198)
Well there is clickhouse. And projects like this https://github.com/flant/loghouse
Re:A log/event processing pipeline you can't have (2019) by user5994461 on Wednesday August 26, 2020 @08:01AM (#24281617)
Well, the article doesn't mention the hardware. Only one machine for 5 TB a day which doesn't add up as far as I am concerned.

If they were storing everything in S3, it's possible to do something similar with ElasticSearch for the same order of costs (maybe three times?), the money going to EBS storage instead. ElasticSearch allows to query and visualize logs, which plain S3 storage doesn't, it's worth a bit more IMO.

Re:A log/event processing pipeline you can't have (2019) by Spivak on Wednesday August 26, 2020 @10:53AM (#24283000)
I think the author's math tracks. Our pipeline looks something like

hosts -> rsyslog collector -> kafka -> custom crunching -> elastic

which is a bit inefficient and we try to go hosts -> kafka when we can but a lot of stuff only supports rsyslog so it's there and simple enough.

The rsyslog collector and our crunchers are teeeny tiny compared to the rest of the pipeline and can chew through up to a week of backlog in a few hours. The bottleneck is the network for us and if we upped the pipe to 10G we could probably get away with a single host.

Re:A log/event processing pipeline you can't have (2019) by cbsmith on Wednesday August 26, 2020 @11:16AM (#24283208)
I missed where a volume of data was described that was outside elastic search's capacity. Seems unlikely that it "would not cope" regardless of configuration.