Usually the most compelling reason not to use a traditional SQL database is that they're expensive to field in distributed environments and require specialist engineers to support when you get out into the weeds.

But I submit it's a category error to call Redis a "NoSQL database." It was also a category error to put data that needs to be redundant and durable into a storage system that is really specialized for high volume but low-reliability cache work.

I mean, seriously. Minimum viable PostgresSQL deploy in EC2 is pretty damn expensive, but you may not have enough cash on hand to buy your own hardware and pay to put it in a "real datacenter." If you can find something cheaper and you're not even seed funded yet because no one funds your social product until you have 100k users, you're probably more likely to actually make it to where you can afford a different database by just using Redis. It's called "technical debt" and we accrue it both bitterly and often willingly.

Lot of people use relational databases as a big key value store, when really they should use a key value store. It's not necessarily about scalability or performance. =)

And similarly, the attitude that a RDBMS is the right and proper data store, that all data should rightly prefer to live in an RDBMS aside from the loftiest heights of scalability.

Finally, the assertion that using a key-value or column-based datastore is going to lead to an "unnecessarily complex" design.

I'd not be surprised to find you disagree, but if we were working together on the same team, these are some things I'd try to work with you on.

Similarly, while it is wise not to lump all of the non-SQL/non-relational databases together, to say that the bulk of them share a common obsession with scalability is not much of a reach. After all, we're not talking about Neo4J, FramerD, MetaKit, etc. We're talking about young systems that were born out of frustration with scalability issues. It's true that they all have different tradeoffs, strengths and weaknesses, but one significant advantage of Postgres is that it does so well in such a variety of situations. Maybe it isn't perfect for every scenario, but it's pretty good for most scenarios. This is partly an inherent advantage of age, and partly the benefit of a sound theoretical foundation. A big thing I dislike about the current crop of NoSQL databases is that you are expected to learn a great deal about their inner workings to make an informed decision about whether or not to use them. MongoDB really epitomizes this. But this is a pretty lazy complaint compared to random data loss and insane defaults (early days of Mongo), and I have to give you that.

Your position sounds quite reasonable compared to the one I've been primed by, but it's going to be a few more years before these databases mature to the point I'd consider one a reasonable first choice for a generic database. I don't draw the line quite as far as the GP—sometimes you have a special case and you need special tools—but I can certainly empathize with questioning the wisdom of making a distributed non-relational database the primary store for a low-load, classic RDBMS problem. But Twilio has come forward and agreed with that position and denying that they do that, so the spark for this flamewar is conjecture. While people have come forward to defend such a configuration, few seem to be endorsing it by using it themselves. This says something about the reasonableness of the idea.

Also might be worth dropping the whole "SQL is better" insinuation. We have seen some pretty major data loss bugs in PostgreSQL and MySQL recently.

When developing such systems, it is irresponsible to use new, unproven technologies without justification.

When developing such systems, it is irresponsible to trade off the reliability and safety of the system for some "developer productivity".

Such irresponsibility is just not acceptable. Failures due to such irresponsibility should not be tolerated, either.

I appreciate that you are very committed to reliability. I also have worked on (telecom) billing systems. However, what are you suggesting here? Should all Twilio's customers find another provider? (Do you have a suggestion?) Should they just not use a service like Twilio's? Should Twilio quadruple their prices so they can throw up enough hardware to do all this in postgres? Should we all fly over to Sicily to protest outside antirez's house?

Incidentally, in the late '90s I sat in on a few sales efforts in which Lucent pushed their "Datablitz" in-memory database for mediation and billing of call records. So, it's not as though nothing similar to this has ever been done.

I believe the engineers at Twilio are building and have built a platform to handle a scale that is going to continue to reach further than many traditional models will scale. It makes sense to me that they would need to look for in memory systems to push the limits - They are fsync'ing immediately - so while it's in memory for a read it's disk for writes so that sounds pretty safe to me... Also, a lot of people use Redis now, so I definitely wouldn't say it's unproven, that is as others have said like saying we should still be using typewriters to put ink on paper because it's "proven"... I'd answer that with... "good luck with that".

And at the end of the day - "the shit will hit the fan" and it did...

the team did great and we're all thankful.

And when we talk about "relational databases", we basically never mean MySQL. I'm not even sure why you'd bother to bring it up, given its poor reputation, its lack of proper functionality in some cases, and the many other (and far better) alternatives.

As for scalability, some other commenter here posted a link to a twitter comment describing the scale in this case: https://twitter.com/dN0t/status/360119871318659074

It clearly states "tens of thousands" of transactions per second. That is well within the capabilities of the lower-end database systems on modern, low-end server hardware. Twilio will require a huge amount of growth just to reach the point where higher end database systems and hardware become appropriate, never mind "a scale that is going to continue to reach further than many traditional models will scale".

Running into significant billing issues due to a likely unnecessary use (in my opinion) of a NoSQL database while still at a relatively small scale is not something that a team should be commended for, and nobody should be "thankful" that the incident did happen.