I'll give you a couple examples. Google stores a huge amount of data in BigTable [1] which is fundamental for the operation of Gmail at scale. Another, Facebook initially used Cassandra [2] for it's Inbox search, then migrated to HBase when their needs changed for their new messaging feature.
Do we know if the the underlying data is stored in RDBMS? I'm guessing not at Google, but yes for Facebook/Inbox search. But my point is I'm not ruling it out, I'm open to different options.
[1] http://en.wikipedia.org/wiki/BigTable [2] http://en.wikipedia.org/wiki/Apache_Cassandra#History
I'm not dismissing the tools out of hand -- I haven't used them except to play around with tutorials. I am pretty sure a lot of people start out with Mongo or whatever without having a use case defined and without having good reasons to eliminate the tried-and-true RDBMS solutions. "Use something cool and new" is not a use case. "I've read on slashdot that SQL databases don't scale" is not a use case.
And yet you feel the need to lecture others on their choices ?
Wow you are something.
http://stackoverflow.com/questions/4255895/why-googles-bigta...
I work on a app that stores documents (PDFs) in Amazon S3 containers, and sometimes those documents get lost or inaccessible for some reason. We have backups for those. The only consistency requirement there is that the PDF opens and the name matches the name in the RDBMS. That's a much lower bar for consistency and durability than our credit card transaction data, for example.
Facebook and Instagram itself I imagine would care VERY much if data went missing as it would mean not just (a) user data going missing but (b) potentially huge inconsistencies in the social graph.
Unless you have some evidence that those companies do not care about data durability ?
"Social graph" - LOL.
Wow.
Whether it is consequential or not is completely beside the point. As is whether you understand the "social graph".
It's not a bank.
By the way, I believe social networking is mostly bullshit, which is my "LOL" at the "social graph".
Facebook has different data stores for each of their features so I imagine the loss of a person would be pretty nasty as would a photo data object that multiple people have tagged or commented on.
My point is that when you are the size of Facebook, Twitter etc a loss of data in 0.001% of cases equates to a LOT of data and relationships.
You don't use Best-guess Attempt at Sorta-consistent Eventualities for important data ;-)
As for this part:
>"When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights."
The obvious answer from the MySQL/NoSQL folks is "What? That's crazy! Multiple apps hitting the same db? Are you nuts???"
See my blog post at http://ledgersmbdev.blogspot.com/2012/09/or-modelling-interl... for more on this mentality.
Thanks for pointing me to your article from the HN flame war about MongoDB. If only there were more useful articles like this and less flaming.
You wrote "...in MySQL, the application is king and the db a humble servant." That is exactly my experience, and it leads not only to terrible databases but to piles of awful code written around terrible databases. In 1974 Kernighan and Plauger wrote "Choose a data representation that makes your program simple" (Elements of Programming Style). Other esteemed programmers have said the same thing. A well-designed and normalized database schema with validations and access rules implemented in the database will reduce the amount of application code needed.
I've written about keeping data management code in the database where it belongs (http://typicalprogrammer.com/?p=101), but you make a much clearer argument with real examples. A few years ago the Ruby on Rails people championed the idea of "application databases" and writing HTTP+XML APIs, which probably seemed like a great idea given that few, if any, Rails applications ever had databases that other applications needed to access. Now the NoSQL folks have taken this even further, eliminating structure from the data and all ACID compliance, separating the data from integrity enforcement, all in the name of scalability problems their app might have if they get as big as Google.
I am stuck with MySQL most of the time because I work with broken web applications, and so far all but a couple have used MySQL. As a long-time Oracle and SQL Server programmer I almost cry trying to figure out some of the MySQL databases I come actoss. I am working on one now that uses GUUIDs as primary keys in every table, generated in PHP code, for reasons I can only attribute to ignorance.
The second point is that I am a relative fan of loosely coupled systems. Loose coupling gets away from CAP theorem problems because each system is autonomous and can always defer interop with other applications until later. On the other hand your centralized RDBMS goes down and every app that needs it dies with it. How many apps we want entangled in that way is a business decision but there are tradeoffs on both sides.
That doesn't mean endorsing the NoSQL approach though. I maintain that you can't have "eventual consistency" in any meaningful, guaranteed way unless you always have "absolute local consistency" within some scope. Postgres-XC defines that scope as cluster-wide. In a loosely coupled system it could be each individual node. For this reason BASE doesn't work for anything that is important. It reduces to:
Best-guess Attempt at Semi-consistent Eventualities.
Lesson is: not only do you want to encapsulate your data, but anywhere that you want eventual consistency, you need to have local ACID compliance for some defined scope of local. This is a big issue that NoSQL db's have and that they cannot solve (due to CAP theorem issues). If only it were absent from the RDBMS world. I groan every time someone asks for "multi-master asynchronous replication."
Not planning for the future tends to do that.
>"I am stuck with MySQL most of the time because I work with broken web applications, and so far all but a couple have used MySQL. As a long-time Oracle and SQL Server programmer I almost cry trying to figure out some of the MySQL databases I come actoss. I am working on one now that uses GUUIDs as primary keys in every table, generated in PHP code, for reasons I can only attribute to ignorance."
Ouch, that sounds painful. I have horror stories I could tell too. Ambiguous foreign keys? ouch.....
I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the team has always been able to figure out a solution without throwing the RDBMS out. One big advantage of working with RDBMSs is the roads are paved and well-mapped. By contrast you still need a machete and good luck in the NoSQL world, and even then you can get sick or lost in the jungle.
I haven't personally worked on a web app with the kinds of database scaling issues I've dealt with in logistics applications involving tens of millions of movements every day.
RDBMSs were invented to allow data malleability. Go back to the old textbooks and look at what preceded the relational model. Remember that data is more valuable and permanent than application code. When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights.