velly

What you do on your "OWN TIME" is your own business. - Mark Zuckerberg

Thanks fresh,

I'd love to get rid of MyISAM. Doing the course of alpha testing it was corrupted once and that never happened with our InnoDB tables. Also, looking at Tokutek as the long term solution for all tables. The jump from InnoDB to Tokutek is straight forward if fulltext is taken out of the mix. I really wish Tokutek biz dev would talk to Amazon about offering the storage engine in RDS. Any who, can you speak to the maintenance profiles of the two? It seems like Sphinx is pretty hands off. Also having many years of Java experience, I feel like Sphinx might be a better bet if I start off running in a memory constrained environment like an EC2 small or medium.

Map/Reduce is a key feature that is required by the "one true datastore" along with the elegance of on the fly indexed views ;)

Bret Taylor of FriendFeed/Facebook fame implemented a scalable system schema free system on top of MySQL where he could attach thin tables arbitrarily to JSON collections as the index. When migrations happened the collections would be scanned and the tables rebuilt. Digg has done similar work, but no open source project ever made the light of day. This schema free system powered most of the site (one true datastore).

After cracking open Redis and then looking at mysql-sr-lib it brought that FriendFeed idea back full circle. Basically I think mysql-sr-lib 2.0 could be a 1:1 implementation of Redis functions implemented as MySQL stored procedures. The only adaptation I would make is that functions would be namespaced (a table per namespace) so that collections could be made. It would be up to the client to decide how to take advantage of namespacing. Do use it for functional collections or sharding for example? Clients would also have to use a distribution strategy to map keys to a namespace/collection. The idea is a lot clearer in my mind now thanks to making the mysql-sr-lib discovery.

From the author:

First off, I apologize for my thoughts being all over the place at 3AM in the morning ;)

I've written several transactional systems using Oracle and DB2 in the financial industry in a former life. The trappings of SQL are directly proportional to it's power. As demonstrated in mysql-sr-lib they can be packaged neatly to great effect however. Also I was talking about transactional operations on datasets.

Lastly, on slowness of SQL. I just meant that query plans cost and are not free. There have been several experiments of late of the amazing performance you can get by working with InnoDB directly instead of going through MySQL for example. Basho is just one of the players that has done work in the space. There is also hook that can be directly embedded in MySQL to bypass the standard workflow and interact directly with the datastore.

My main point in all this is that it will take some time to reach a 1.0 version of a viable, modern persistence solution in Redis. Time would be better spent now at least evaluating other open source persistent engines before diving head first into a very shallow pool. The main point is that most of the current persistence tools have been battle tested in a number of environments, antirez should leverage that considering the small size of his team; just two people.

Bingo! (from the author). I wrote this late at night at the end of a long day, but that's the point.

This was a just a quick blurb on my thoughts around HTML5 and how it impacts mobile from the perspective of the big boys. I definitely think that Facebook and Google come out stronger if the web and it's links keep working the way they do now. The nodes at the end of those links need to be more capable, but the web as a network is what needs to remain strong and open. The one ruler model that Apple has won't scale. The financial markets for one, from angel investors up to the public markets won't tolerate a 30% tax on innovation.