News for hackers. Stuff that matters.
Re:Reaching 200K events/sec by aphyr on Thursday January 31, 2013 @08:44PM (#5148680)
If you're wondering about the workload, this is the trivial Riemann config this benchmark uses:

  (streams
    (rate 5 (comp prn float :metric))
Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console.

I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize those layers--it's not meant to stress the internal stream library, the state index, or pubsub. When I start optimizing those components, I'll have more "real-world" numbers to report.

I should also explain that this particular post explores the high-throughput, high-latency range of the client spectrum. End-to-end TCP latencies (not counting wire time) for single-event messages are on the order of ~100 microseconds-1ms, with occasional spikes to ~30ms depending on JVM GC behavior.

Re:Reaching 200K events/sec by spullara on Thursday January 31, 2013 @09:44PM (#5148885)
You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.
Re:Reaching 200K events/sec by aphyr on Friday February 01, 2013 @01:17AM (#5149518)
Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :)

The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this test. The client is a total hack using Java OIO, calling flush() on every message, Nagle's algorithm disabled for low per-msg latencies... it's a wreck, haha. Part of next week's optimization push is replacing it with a Netty client and tuning TCP options for various workloads.

Re:Reaching 200K events/sec by no7hing on Friday February 01, 2013 @04:46AM (#5150010)
Things will probably look better once you switch your client implementation to Netty as a similar test I did with Netty 4, Protobufs and batching of messages produced low seven figure numbers too.
Re:Reaching 200K events/sec by spullara on Friday February 01, 2013 @01:10PM (#5152319)
Ah ok, my test was run using the C redis-benchmark client.
Re:Reaching 200K events/sec by thebluesky on Friday February 01, 2013 @06:08AM (#5150191)
Is the source available anywhere?

1.5M locally or over a network? What is the size of each message?

Re:Reaching 200K events/sec by spullara on Friday February 01, 2013 @01:09PM (#5152311)
http://github.com/spullara/redis-protocol — server module

The test was run with "SET key value" which in the redis protocol is something like 15-20 bytes / message.