Re:Reaching 200K events/sec
by aphyr on Friday February 01, 2013 @01:17AM
(#5149518)
Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :)
The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this test. The client is a total hack using Java OIO, calling flush() on every message, Nagle's algorithm disabled for low per-msg latencies... it's a wreck, haha. Part of next week's optimization push is replacing it with a Netty client and tuning TCP options for various workloads.
Re:Reaching 200K events/sec
by no7hing on Friday February 01, 2013 @04:46AM
(#5150010)
Things will probably look better once you switch your client implementation to Netty as a similar test I did with Netty 4, Protobufs and batching of messages produced low seven figure numbers too.
The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this test. The client is a total hack using Java OIO, calling flush() on every message, Nagle's algorithm disabled for low per-msg latencies... it's a wreck, haha. Part of next week's optimization push is replacing it with a Netty client and tuning TCP options for various workloads.