News for hackers. Stuff that matters.
Re:From Kafka to ZeroMQ for real-time log aggregation by agentgt on Saturday September 17, 2016 @08:00AM (#12520159)
I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics.

And our log messages are ridiculously big at times (15k to as big as 50k).

Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k messages/s when embarrassingly had debug turned on in production and RabbitMQ did not fail but Elastic Search and sadly Flume did as well (I tried to get rid of flume with a custom Rust AMQP to Elastic Search client but at the time had some bugs with the libraries.. Maybe I will recheck out Mozilla Heka someday).

There is this sort of beating of the developer chest with a lot of tech companies.. that hey listen we are ultra important and we are dealing with ridiculously traffic and we need ultra high performance. Please tell/show me these numbers.... Or maybe stop logging crap you don't need to log.

Or maybe I'm wrong and we should log absolutely everything and Auth0 made the right choice given their needs (lets assume they have millions of messages a second), I still think I could make a sharded RabbitMQ go pretty far.

This goes with other technology as well. You don't need to pick hot glamorous NoSQL when Postgresql or MySQL and a tiny bit of engineering will get the job done just fine particularly when mature solutions give you such many things free out of the box (RabbitMQ gives you a ton of stuff like a cool admin UI and routing that you would have to build in ZeroMQ).

Re:From Kafka to ZeroMQ for real-time log aggregation by packetized on Saturday September 17, 2016 @10:14AM (#12520553)
We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.
Re:From Kafka to ZeroMQ for real-time log aggregation by agentgt on Saturday September 17, 2016 @10:29AM (#12520613)
I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace.

BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it.

In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).

Re:From Kafka to ZeroMQ for real-time log aggregation by gerakinis on Saturday September 17, 2016 @02:17PM (#12521669)
You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)
Re:From Kafka to ZeroMQ for real-time log aggregation by trimbo on Saturday September 17, 2016 @11:09AM (#12520781)
> cluster failing with some network issues we had with Rackspace

Were you using Zen Discovery at the time?

I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.

Re:From Kafka to ZeroMQ for real-time log aggregation by gerakinis on Saturday September 17, 2016 @02:19PM (#12521677)
Are you using HA functionality and also on disk backing? These two things bring down performance roughly 5-10x and are mostly required for situations that can't afford message loss. I still like the rabbitmq solution, it is my own, but i've found it takes more hardware than you are suggesting.
Re:From Kafka to ZeroMQ for real-time log aggregation by smetj on Saturday September 17, 2016 @03:31PM (#12522020)
Yes that's my experience as well.
Re:From Kafka to ZeroMQ for real-time log aggregation by trimbo on Saturday September 17, 2016 @11:16AM (#12520812)
> We run an average of 14k logs/sec through a two-node RMQ cluster

How many MB/s are you indexing?

Re:From Kafka to ZeroMQ for real-time log aggregation by benmccann on Saturday September 17, 2016 @11:59AM (#12521025)
What are the new options that help improve indexing rate?
Re:From Kafka to ZeroMQ for real-time log aggregation by smetj on Saturday September 17, 2016 @03:29PM (#12522013)
100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s?

I'd be really interested to hear how you can achieve such a thoughput with rabbitmq

Re:From Kafka to ZeroMQ for real-time log aggregation by agentgt on Sunday September 18, 2016 @09:19AM (#12525167)
I'm not sure how much I can help because I didn't setup the RMQ cluster so I don't know the configuration details but I know it is fairly powerful (its also partly why I can't entirely be critical of Auth0 because ZMQ is probably far less expensive infrastructure wise).

I do know we use multiple queues and even exchanges (and I did not know about the one core to queue).

A simple googling shows though folks have achieved far greater throughput[1] than 100k (and by the way this wasn't sustained.. it was spikes).

[1]: https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-m...

Re:From Kafka to ZeroMQ for real-time log aggregation by gerakinis on Saturday September 17, 2016 @02:23PM (#12521689)
Heka is really stupid to configure, really fast, and now deprecated... I've used it for log tailing and metrics forwarding extensively and can't recommend it enough if you need to use amqps out.

If you don't need amqps out there are more modern, better supported projects.

Re:From Kafka to ZeroMQ for real-time log aggregation by brightball on Saturday September 17, 2016 @08:47AM (#12520271)
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
Re:From Kafka to ZeroMQ for real-time log aggregation by agentgt on Saturday September 17, 2016 @09:00AM (#12520305)
Yep... I was the first [email protected] circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
Re:From Kafka to ZeroMQ for real-time log aggregation by brightball on Saturday September 17, 2016 @09:38AM (#12520429)
No apologies needed, lol. I went to Clemson. Just saw it and was curious.