Wallaroo: a fast, scalable data processing engine

Re:Wallaroo: a fast, scalable data processing engine by snowAbstraction on Thursday September 28, 2017 @03:04PM (#15359861)

Thanks for your answers.

If you feel up tackling my first question, I have tried to clarify it:

I was referring to statistical variance of latencies listed in the blog post: 50% 95% 99% 99.9% 99.99% 66µs .5ms .5ms .5ms 1 ms

More concretely, what is the difference between when ingesting and processing an event takes 1 ms and it takes 66µs? Do you know what causes this?

[ Parent | on HN ]

Re:Wallaroo: a fast, scalable data processing engine by spooneybarger on Thursday September 28, 2017 @05:47PM (#15361020)

We read in large chunks of messages at a time and start processing them. Some messages wait longer in internal queues for CPU time to become available to process them.

The benchmark we use is intentionally very light computationally. We did that because, if we had a heavy computation then the overhead from Wallaroo would be hidden.

[ Parent | on HN ]