News for hackers. Stuff that matters.
Re:Internal Design of Onyx by sixdimensional on Thursday October 09, 2014 @01:33PM (#8433864)
I'm interested in this general approach being implemented in so many newer engines (those which Onyx apparently competes with) - the distributed pipeline processing/directed acyclic graph (DAG) approach. So many of these engines have related components - message queues, central coordinator, etc.

It seems this is the current popular technique (and rising in popularity) to solving some really long-standing, hard, data processing problems and related performance concerns - really an extension beyond the progress we made with distributed MapReduce algorithms and the like in Hadoop.

There are certainly a lot of implementations coming out - why would one choose Onyx over say, something like Spark? Is it just the compatibility with the Clojure programming language and the associated, implied benefits of functional programming paradigms (among others?).

Re:Internal Design of Onyx by XPherior on Thursday October 09, 2014 @01:47PM (#8433954)
Hello! Michael Drogalis, the developer, here.

I talked a little bit about why I wrote Onyx on the Clojure mailing list: https://groups.google.com/d/msg/clojure/OmHzAEfYe9U/33e0a0l3...

It's mostly driven by the need to use data structures in places where I didn't have them - I only had things like macros and functional composition.

Re:Internal Design of Onyx by sixdimensional on Thursday October 09, 2014 @05:39PM (#8435150)
Thanks for the info, and awesome work on Onyx!
Re:Internal Design of Onyx by sixdimensional on Thursday October 09, 2014 @01:42PM (#8433921)
More information for those interested from the developer's tumblr: http://michaeldrogalis.tumblr.com/post/98378329045/onyx-a-ne...
Re:Internal Design of Onyx by bkirwi on Thursday October 09, 2014 @02:08PM (#8434091)
Very interesting! I'm working on a similar project at the moment, but I haven't come across Onyx at all before -- I'll start digging through.

In particular, I'm interested to know if this supports exactly-once messaging semantics, and if so, what the implementation looks like. Anyone here have experience with this?

Re:Internal Design of Onyx by XPherior on Thursday October 09, 2014 @02:15PM (#8434139)
I'm not sure if exactly-once messaging is possible in a distributed system. Onyx gets close in that it uses transactions to move data across queues atomically. But code can fail just before a transaction is committed. The transaction will only be committed once, but the code leading up to the transaction will be run twice. Is that really exactly-once? IMO, it's not.
Re:Internal Design of Onyx by bkirwi on Thursday October 09, 2014 @03:22PM (#8434511)
Right: I guess I distinguish between exactly-once _messaging_ and exactly-once _processing_. it doesn't seem possible to guarantee that the code will be run exactly once, but you can promise that the outputs are made visible exactly once... as long as your system can capture the relevant outputs, of course.

It seems like transactions ought to be enough for the latter -- I'll have a look. Thanks!

Re:Internal Design of Onyx by angersock on Thursday October 09, 2014 @03:56PM (#8434684)
I've sent a link to this out to my team as an example of the sort of documentation and technical writing I enjoy. Go one level up in the directory and you'll see quite good explanations of deployment practices, implementation, and high-level architecture stuff.
Re:Internal Design of Onyx by XPherior on Thursday October 09, 2014 @04:07PM (#8434742)
Thank you. :)