Great paper. I tend to work on medium scale systems but I always enjoy reading material from companies like Google, Facebook and Twitter on how they manage scaling, especially data stores.

There is an interesting pattern: the more scalable a data store is the larger the hassle of writing application code against it. I am lucky that MongoDB master slave has been sufficient for what I have been doing because it is almost as simple to use as a RDMS. It is clear that writing against HBase and Cassandra takes more effort and it is more difficult to be in an agile development mode.

The architecture of MegaStore is so complicated... There are so many roles in the system, e.g. stateless replicate server, different stateful paxos instances, stateful co-coordinator, bigtable and other auxilary roles.

I suspect some roles are added later because the original implementation does not meet their design goal, as patch to normal software product.

It is a nightmare for operation engineers of GAE.