News for hackers. Stuff that matters.
Re:Onyx: fault tolerant data processing for Clojure by dj-wonk on Friday March 06, 2015 @12:53PM (#9158389)
Re: Onyx's architecture. I would wonder about performance when keeping a shared log in ZooKeeper. Why not use something like Kafka -- it is designed for high-volume, immutable logging. ZK works best for less-frequently changing configuration, such as node connection information or snapshotting. I could be wrong. I'd like to hear your thoughts and experience.
Re:Onyx: fault tolerant data processing for Clojure by XPherior on Friday March 06, 2015 @12:56PM (#9158404)
- Picking up Kafka means introducing another dependency.

- Onyx's log doesn't grow particularly large because it's only used for coordination, not for messaging.

- Because the log isn't huge, and can be GC'ed, consumers don't experience high volumes of messages.

- ZooKeeper offers sequential node creation - making it a really good fit for what the log needs to do.