News for hackers. Stuff that matters.
Re:Wallaroo: a fast, scalable data processing engine by makmanalp on Wednesday September 27, 2017 @07:01PM (#15353134)
So are we talking about something like a compile-to-native-binary storm / kafka?
Re:Wallaroo: a fast, scalable data processing engine by spooneybarger on Wednesday September 27, 2017 @07:17PM (#15353236)
The C++ API that we currently aren't supporting and the Go language bindings that we are working on create native binaries; yes.

The Python version embeds a Python 2.7 interpreter inside a running application. We have C code that acts as an interface between our Pony runtime and the developer's Python code, calling the required framework functions at the correct time.

The event-by-event features in Wallaroo are somewhat analagous to Storm but we had management of in-memory state that allows you to grow the size of the cluster and have the application state redistributed across the cluster. This can be done while an applicaiton is running. In the case of Storm, this would be like:

1- keeping your application state in objects in your bolts (this helps with performance compared to an external store) 2- having a means to add more workers to the running storm topology and have that in bolt state redistributed across the new number of machines.

The 15-minute "scale independence" video shows this in action: https://vimeo.com/234753585

In many ways, Wallaroo looks a lot like the systems that Pat Helland describes in "Beyond Distributed Transactions": http://queue.acm.org/detail.cfm?id=3025012

One of our engineers is giving a talk at Velocity conf NY next week. He's going to be discussing Pat's paper and how Wallaroo exhibits a number of characteristics he describes.

Re:Wallaroo: a fast, scalable data processing engine by makmanalp on Wednesday September 27, 2017 @07:39PM (#15353336)
Yes, that paper reference was exactly what I wanted. Thank you!
Re:Wallaroo: a fast, scalable data processing engine by spooneybarger on Wednesday September 27, 2017 @07:45PM (#15353370)
If you are into papers and talks, we have a few more on the website that we love to turn people on to:

http://www.wallaroolabs.com/community/talks

Re:Wallaroo: a fast, scalable data processing engine by makmanalp on Wednesday September 27, 2017 @08:15PM (#15353501)
Brilliant! I've already read the fault injection one, excited to look at the others!
Re:Wallaroo: a fast, scalable data processing engine by spooneybarger on Thursday September 28, 2017 @08:58AM (#15356731)
I love Peter's work. I gave a talk on fault injection and how we use it to test Wallaroo:

https://www.monkeysnatchbanana.com/2016/11/22/codemeshio-how...