News for hackers. Stuff that matters.
Re:Apache Kafka 0.8.0 released by nullymcnull on Monday December 09, 2013 @10:12AM (#6874609)
Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?
Re:Apache Kafka 0.8.0 released by jbooth on Monday December 09, 2013 @11:45AM (#6875095)
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".

Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.

So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).

Re:Apache Kafka 0.8.0 released by xorgar831 on Monday December 09, 2013 @01:47PM (#6875923)
How does that compare to RabbitMQ's disk backed store?
Re:Apache Kafka 0.8.0 released by jbooth on Monday December 09, 2013 @06:30PM (#6877854)
I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
Re:Apache Kafka 0.8.0 released by whisk3rs on Monday December 09, 2013 @10:34PM (#6878825)
It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
Re:Apache Kafka 0.8.0 released by recuter on Monday December 09, 2013 @02:20PM (#6876166)
How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?
Re:Apache Kafka 0.8.0 released by ddorian43 on Tuesday December 10, 2013 @05:42AM (#6879963)
Since you host it yourself you can't compare it because you on the variable cost of servers and you time to maintain them.
Re:Apache Kafka 0.8.0 released by ecoffey on Monday December 09, 2013 @11:07AM (#6874888)
In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream.

EDIT: should add that morkbot had a great link too:

https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...

Re:Apache Kafka 0.8.0 released by RyanZAG on Monday December 09, 2013 @10:59AM (#6874837)
Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.
Re:Apache Kafka 0.8.0 released by idunno246 on Monday December 09, 2013 @12:04PM (#6875198)
It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.
Re:Apache Kafka 0.8.0 released by boredandroid on Monday December 09, 2013 @12:52PM (#6875542)
Re:Apache Kafka 0.8.0 released by idunno246 on Monday December 09, 2013 @02:32PM (#6876271)
Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires."

Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc.

It's still very good at what it does.

Re:Apache Kafka 0.8.0 released by mumrah on Monday December 09, 2013 @03:38PM (#6876741)
Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8.

What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.

Re:Apache Kafka 0.8.0 released by mumrah on Monday December 09, 2013 @03:46PM (#6876821)
Use cases (from the project page): http://kafka.apache.org/documentation.html#uses