News for hackers. Stuff that matters.
Re:Scaling Riak at Kiip by tptacek on Thursday May 24, 2012 @05:12PM (#4020589)
It's been a riak-y day for me, so I watched this avidly.

You're no doubt wondering what the issues were; they come 20 minutes in, and were (paraphrased):

* At scale in production, adding a new node took days to complete all the handoffs; they recommend adding new nodes as soon as it's looking like you need them, rather than waiting until you're redlining.

* 2i is slow, especially in EC2; a straight KV "get" is milliseconds-denominated; 2i index queries were taking multiple seconds. Use 2i, they say, but in background processes.

* Javascript MapReduce is slow; this is well known. They confirm Erlang MR was adequate.

* As the LevelDB keyspace grows, there's a stepping function in latency; 5ms, then 15ms, then 25ms; the solution is to add nodes. (LevelDB is Google's KV store, a new option for Riak, required if you're using secondary indexes).

* Riak Control didn't work for them over low-latency connections.

* Once, a Chef misconfiguration left the whole cluster flapping on and off, which corrupted the cluster; they recovered with Basho support. Be careful about adding and removing nodes rapidly.

* Similarly, flapping a single node caused the cluster to get into a state where it wouldn't converge again; the cluster worked but no nodes could be added until they (presumably?) restarted it.

Re:Scaling Riak at Kiip by dsl on Thursday May 24, 2012 @06:37PM (#4020959)
After watching the video, the takeaway for me was don't run $database on EC2 instances.

Last night I just finished replacing 6 maxed out medium instances with one $100 box from SoftLayer. :/

Re:Scaling Riak at Kiip by bravura on Thursday May 24, 2012 @08:01PM (#4021226)
At that price point, it seems that you can get FAR better specs on Honelive. Kimsufi.ie is even cheaper, if you don't mind your server being hosted in the UK.

edit:

Softlayer: Intel 4x2.40GHz, 2 GB RAM ECC, $160/mo Honelive: Intel i7-2600 16, 16 GB RAM, $120/mo Kimsufi.ie: Intel i7 4x 2(HT)x 2.66+ GHz, 24 GB, $60/mo

Re:Scaling Riak at Kiip by moonboots on Thursday May 24, 2012 @08:19PM (#4021279)
Have you heard any reviews of Honelive? I've been looking for a dedicated server provider in the US with comparable prices to Hetzner in Germany.
Re:Scaling Riak at Kiip by dsl on Thursday May 24, 2012 @08:35PM (#4021319)
I get a lot of nice little extras from SoftLayer. For example I can get boxes in the 5 corners of the internet (west, central, east, euro, asia) with free bandwidth between them.
Re:Scaling Riak at Kiip by StavrosK on Friday May 25, 2012 @09:59AM (#4023469)
Has anyone used Kimsufi? Are they any good? I'm at Hetzner right now, but I never found any use for that box, so I'm just paying for it for nothing...
Re:Scaling Riak at Kiip by bravura on Friday May 25, 2012 @12:16PM (#4024138)
I just signed up for kimsufi. They are the budget brand of OVH. Webhostingtalk gives them favorable reviews.
Re:Scaling Riak at Kiip by spudlyo on Thursday May 24, 2012 @07:14PM (#4021064)
And if you do, for God's sake don't run them on EBS volumes. If you don't know by now that EBS performance is highly variable, you really haven't been paying attention.
Re:Scaling Riak at Kiip by j2labs on Thursday May 24, 2012 @06:04PM (#4020828)
Very useful data. Do you have anything more comprehensive about your experience?
Re:Scaling Riak at Kiip by tptacek on Thursday May 24, 2012 @06:18PM (#4020883)
I don't work at Kiip. :)
Re:Scaling Riak at Kiip by j2labs on Thursday May 24, 2012 @06:21PM (#4020899)
I thought you meant to say you had additional personal experience, but perhaps you've just been reading about Riak today.

Thanks anyway.

Re:Scaling Riak at Kiip by tptacek on Thursday May 24, 2012 @08:28PM (#4021303)
I've had a 3-node hardware Riak cluster lying around for about 4 months now, but only just today cut over from Postgres to it. I can talk to you about how shiny Riak is on day 1, but as this presentation shows, the day 1 behavior of a Riak cluster can be a bit of a trap.
Re:Scaling Riak at Kiip by gregburd on Friday May 25, 2012 @02:39AM (#4022238)
Great to hear that you're cutting over to Riak. We highly recommend that your minimum cluster size is your N value (replication value) + 2. I your case that likely means 5 nodes for your starting cluster, not 3. There are many reasons. http://basho.com/blog/technical/2012/04/27/Why-Your-Riak-Clu...
Re:Scaling Riak at Kiip by tptacek on Friday May 25, 2012 @06:26AM (#4022865)
Obviously great to know. Thanks!
Re:Scaling Riak at Kiip by saturdayplace on Thursday May 24, 2012 @11:39PM (#4021780)
Thomas, I'm in the middle of evaluating document stores and am perhaps getting stars in my eyes from Riak's easy-to-scale story. I'd love to hear more about your experiences once your cluster has had time to marinate a bit.
Re:Scaling Riak at Kiip by mrkurt on Thursday May 24, 2012 @08:31PM (#4021309)
Interestingly enough, this is a very similar type list as you'd see for Mongo, especially in terms of overall effort.

That you'd encounter all these things at a 25mm daily ops level is pretty odd, though.

Re:Scaling Riak at Kiip by rogerbinns on Thursday May 24, 2012 @09:20PM (#4021432)
The very same folks did write about their MongoDB experience: http://blog.engineering.kiip.me/post/20988881092/a-year-with...
Re:Scaling Riak at Kiip by jasonmccay on Thursday May 24, 2012 @10:11PM (#4021557)
I think the point he was making was that the company in question never tried to scale horizontally with MongoDB because, in their words, "we believe horizontally scaling shouldn’t be necessary for the relatively small amount of ops per second we were sending to MongoDB."

Yet, they went and scaled horizontally with Riak and experienced pain.

Their opinion that it did not make sense to have to horizontally scale the "relatively small" number of ops they were sending to MongoDB is certainly their own, but then they horizontally scaled with Riak anyway and boasted about their 25MM ops per day scaling ... which, averaged out, is only about 280 ops per second.

In short, it was far from an apples to apples comparison.

Re:Scaling Riak at Kiip by tptacek on Thursday May 24, 2012 @10:32PM (#4021612)
It's probably significantly more than 280 ops/sec, given peak times.
Re:Scaling Riak at Kiip by Cloven on Friday May 25, 2012 @02:43PM (#4024872)
"At scale in production, adding a new node took days to complete all the handoffs..."

That's a bit of a headscratcher. What is happening during those 'days' and what is the primary limiting factor?

I keep meaning to get into Riak, but then stuff like this where the system has crazy moments that are impossible to coherently reason about keep popping up.

Re:Scaling Riak at Kiip by tptacek on Saturday May 26, 2012 @01:31PM (#4027800)
It's not a crazy moment. They have a system running at real scale, and they found that while keeping the system up constantly with immense amounts of data in it, they were able to dynamically add a new node to their cluster --- just that balancing everything out took a lot of time for the system.

The operational challenge I infer from this is that they had waited to add that node until they really needed it, because their expectation was that adding the node would get them quick relief to their scaling issue. Instead, they got relief a few days later when the node was fully integrated.

Solution: don't wait to add nodes until the last minute.