2. We measure from the moment that we ingest the message to when the message is done being processed. If a Wallarooa application was receiving data over TCP and sending out new data also over TCP, it would be from the time that we read in the data to when we write the message out onto the outgoing socket. If a message were to be filtered and not creaye any output then the end point would be once its filtered.
3. We currently use TCP between nodes. The latency that gets added with a network hop in part depends on how much traffic is going over that hop. If you were processing a couple hundred thousand messages or so a seond, that additional latency is general 500 micros to 2 milliseconds.
4. I suppose. I don't believe anyone has tried.
The metrics that we provide include not just end to end latencies but also the amount of time spent in a given computation, going over a given network hop and similiar insight.
Wallaroo has a fairly low overhead so most time in an application is spent in user computations (the latency of which Wallaroo can't control) and any network hops.
Thank you!
If you feel up tackling my first question, I have tried to clarify it:
I was referring to statistical variance of latencies listed in the blog post: 50% 95% 99% 99.9% 99.99% 66µs .5ms .5ms .5ms 1 ms
More concretely, what is the difference between when ingesting and processing an event takes 1 ms and it takes 66µs? Do you know what causes this?
The benchmark we use is intentionally very light computationally. We did that because, if we had a heavy computation then the overhead from Wallaroo would be hidden.
1. Where is the latency variance coming from? (Garbage collection? Network stack?) 2. Between which two points are you measuring it? 3. What would the variance look like if using two processing nodes on different machines? 4. Is it possible to use something like LTTng with Pony to understand latency distribution better?
All-in-all a pretty cool project!