We are first evaluating MongoDB. I believe the main reason behind this is we are already using Mongo in other parts of our application, so there is no additional setup when converting.

Note that nothing is set in stone. The decision to begin migrations only happened today. It is possible that we will end up using some other technology altogether, or even we find out the issues we are having with Aerospike and continue using that service.

oh... dude... exact same mistake twice...

OK, let me try to be more constructive. Since accounts are independent, shard based on account (in the application, not in some magic shard-distributing layer). Treat each shard as its own cluster.

If you want super fast requests but can accept being down for an hour or two a couple times a year, a shard can be a single beefy host with a replicating slave. I'd consider either Redis or Mysql/Postgresql. Really, these old-style sql databases can be the fastest things that have the kind of consistency you need.

I've maintained a mongodb cluster configured a couple of different ways. Performance at high load and reasonable consistency is not as great as some older alternatives.

You should take a look at "Call me maybe: MongoDB" [0]

[0] https://aphyr.com/posts/284-call-me-maybe-mongodb

Not to mention "Call me maybe: Redis" [0]. Rinse and repeat for every NoSQL database out there.

Point is, @aphyr skewers everybody. Aerospike is just the flavor of the month.

[0] https://aphyr.com/posts/283-call-me-maybe-redis

I think FoundationDb got great scores from Aphyr, sadly it is now owned by Apple. For the others the big problem is when they promise some kind of ACID and cannot achieve it, if they were explicit with what the supported all customers could make informed decisions. Relations systems are generally bad ad horizontal scaling and can get very slow with full ACID over many servers.

> I think FoundationDb got great scores from Aphyr,

FoundationDB ran Jepsen internally and reported stuff[0], Kyle never worked with it.

[0] http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j... half broken now, none of the images load for me. @aphyr seems to have taken them at their word wrt testing though: https://twitter.com/aphyr/status/405017101804396546

Curious: how difficult is it to replicate aphyr's tests? Can we beleive independent entities posting jepsen reports on some random DB?

I think that's a false equivalence. Aphyr is pretty positive about Riak (with the correct configuration) and Cassandra (when used for appropriate scenarios). If I was choosing a new system those are the two I'd be looking at.

Antirez was very receptive http://antirez.com/news/55

Aphyr wasn't overly impressed by that response though https://aphyr.com/posts/287-asynchronous-replication-with-fa...

Try HBase. At another AdTech company, we have been very happy with it. https://eng.yammer.com/call-me-maybe-hbase

Nooooooooooooooooooooo! Seriously, no! Use mongoDB, PostgreSQL, even flat files if you must, but HBase?

We used it in production about 3-4 years ago and it was a nightmare from both usage and especially maintenance point. Fortunately we had a flat-files based backup system so we were able to rescue data every! Single! Time! the damn thing crashed and took (part of) data with it.

Of course, this is anecdotal evidence, and things might have changed from then, but I wouldn't touch it. Life is too short.

EDIT: Also, I am curious how the results in the above link would compare to aphyr's if he performed the test on HBase?

I see where you are coming from. HBase was unstable 3-4 years ago, but after a great amount of dev effort and battle hardening from Cloudera, Salesforce, etc., it is very stable now. We have ~ 400 nodes running in production for a very critical use case and have seen 0 data loss edge cases in the last 2 years, along with some of our servers running > 6 months without any reboots.

We use is in a very real time use case with latency requirements of single digit milliseconds, and if you tweak it the right way, you can the required performance from it, along with easy horizontal scaling.

Also, I am curious too for aphyr to take on HBase, but I don't think the result would be different since running Jepsen is straightfoward and not much to a person's interpretation. The results and further experiments are what aphyr does nicely.

Thanks for the info on HBase stability. I probably won't use it again (once burnt...), but if they really managed to pull their act together - good for them!