News for hackers. Stuff that matters.
Re:Apache Kafka 0.8.0 released by idunno246 on Monday December 09, 2013 @12:04PM (#6875198)
It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.
Re:Apache Kafka 0.8.0 released by boredandroid on Monday December 09, 2013 @12:52PM (#6875542)
Re:Apache Kafka 0.8.0 released by idunno246 on Monday December 09, 2013 @02:32PM (#6876271)
Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires."

Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc.

It's still very good at what it does.

Re:Apache Kafka 0.8.0 released by mumrah on Monday December 09, 2013 @03:38PM (#6876741)
Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8.

What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.