Re:From Kafka to ZeroMQ for real-time log aggregation
by packetized on Saturday September 17, 2016 @10:14AM
(#12520553)
We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.
[ Parent
| on HN ]
Re:From Kafka to ZeroMQ for real-time log aggregation
by agentgt on Saturday September 17, 2016 @10:29AM
(#12520613)
I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace.
BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it.
In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).
Re:From Kafka to ZeroMQ for real-time log aggregation
by gerakinis on Saturday September 17, 2016 @02:17PM
(#12521669)
You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)
Re:From Kafka to ZeroMQ for real-time log aggregation
by trimbo on Saturday September 17, 2016 @11:09AM
(#12520781)
> cluster failing with some network issues we had with Rackspace
Were you using Zen Discovery at the time?
I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.
Re:From Kafka to ZeroMQ for real-time log aggregation
by gerakinis on Saturday September 17, 2016 @02:19PM
(#12521677)
Are you using HA functionality and also on disk backing? These two things bring down performance roughly 5-10x and are mostly required for situations that can't afford message loss. I still like the rabbitmq solution, it is my own, but i've found it takes more hardware than you are suggesting.
Re:From Kafka to ZeroMQ for real-time log aggregation
by smetj on Saturday September 17, 2016 @03:31PM
(#12522020)
Yes that's my experience as well.