>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version

Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev environment but in production it would lead to timeouts in requests for the duration of the migration for large clients (our application is sharded per tenant) because the table was very large for some of them and the migration locked it. The staging environment helped us find the problem before hitting production because we routinely clone production data (deanonymized) of the largest tenants to find problems like this. It's not practical (and maybe not very legal too) to force every developer have an up-to-date copy of that database on every VM/laptop, and load tests in an environment very similar to production show more meaningful results overall. And feature flags wouldn't help either because they only guard code. So far I'm unconvinced, it sounds pretty risky to me to go straight to prod.

I agree however that the concern about conflicts between feature toggles is largely a monolith problem, it's a communication problem when many teams make changes to the same codebase and are unaware of what the other teams are doing.

> Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

This is definitely best practice, but it's not strictly necessary if a small amount of downtime is acceptable. We only have customers in one timezone and minimal traffic overnight, so we have quite a lot of leeway with this. Frankly even during business hours small amounts of downtime (e.g. 5 minutes) would be well tolerated: it's a lot better than most of the other services they are used to using anyway.

> Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

This doesn't have to be true. You can create an entirely separate table with the new data. New code knows how to join on this table, old code doesn't and thus ignores the new data. It doesn't work for every kind of migration, but in my experience, it's preferred by some DBAs if you have billions and billions of rows.

Example: `select user_id, coalesce(new_col2, old_col2) as maybe_new_data, new_col3 as new_data from old_table left join new_table using user_id limit 1`

For us we use pt-online-schema-change which copies the table first, sets up some triggers to keep things synced, then renames the tables at the end.

> because we routinely clone production data (deanonymized)

Are you using an external service or in-house tool to perform this operation?

I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

Put the DB on docker (or provide some other one touch way to install a clean database). Run all migration scripts to get a current schema, insert sample data, now do your testing. Then make this part of the build process. Then, be sure that you detect after the migrations that the regression tests are failing and prevent merge. The key is having a DB that can be recreated as a nearly an atomic operation.

I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.

> Otherwise it wouldn't be possible to have migrations that work with both old and new code.

Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction:

1. rename the original table "out of the way" of the old code

2. add a new column of the new type

3. run an "INSERT ... SELECT ..." to populate the new column from a transformation of existing data

4. drop the old column of the old type

5. rename the new column to the old column's name

6. define a view with the name of the original table, that just queries through to the new (original + renamed + modified) table for most of the original columns, but which continues to serve the no-longer-existing column with its previous value, by computing its old-type value from its new-type value (+ data in other columns, if necessary.)

Then either make sure that the new code is reading directly from the new table; or create a trivial passthrough view for the new version to use as well.

(IMHO, as long as you've got writable-view support, every application-visible "table" should really just be a view, with its name suffixed with the ABI-major-compatibility-version of the application using it. Then the infrastructure team — and more specifically, a DBA, if you've got one — can do whatever they like with the underlying tables: refactoring them, partitioning them, moving them to other shards and forwarding them, etc. As long as all the views still work, and still produce the same query results, it doesn't matter what's underneath them.)

Not to be rude but this isn't how this works at all. Things like 'run an "INSERT ... SELECT ..."' can't happen at scale due to locking. How they actually do it is super rad:

https://www.percona.com/doc/percona-toolkit/3.0/pt-online-sc...

tl;dr; They setup a system of triggers (updates,inserts,etc) , copy the data over, then run through all the data in the trigger system. percona developed all these fancy features as well to monitor replica data etc. Another way with cloud vms (terabyte+ tables), you image a replica, do the alter, let the replica catch up, image it, build replicas off that, promote this to master.

Facebook's internal one I hear is close to this.

"This isn't how things work at all" implies that more than 0.0000001% of DBs are running "at scale." Most DBs people will ever deal with — including within big enterprises! — are things like customer databases that have maybe 100k records in them. The tables lock waiting for lower-xact-ID read traffic to clear, yes — for about 400ms. Because queries running on the DB are all point queries that don't run any longer than that. Or they're scheduled batch jobs that happen during a maintenance window.

At scale, you're hopefully not using an RDBMS as a source of truth in the first place; but rather, using it as a CQRS/ES aggregate, downstream of a primary event store (itself likely some combination of a durable message-queue, and archival object storage for compacted event segments.) In that kind of setup, data migrations aren't the job of the RDBMS itself, but rather the job of the CQRS/ES framework — which can simply be taught a new aggregate that computes a single new column, and will start catching that aggregate up and making the data from it available. If your RDBMS is columnar (again, hopefully), you can just begin loading that new column in, no locking on the rest of the table required.

IMHO, the trigger-based approach is a weak approximation of having a pre-normalization primary source. It's fine if you want to avoid rearchitecture, but in a HOLAP system (which is usually inevitable for these sorts of systems when their data architects have tried to focus on simplicity, as this leads to eschewing denormalized secondary representations in OLAP-focused stores) it will cause your write perf to degrade + bottleneck, which is actually the worst thing for locking.

(I should know; I'm dealing with a HOLAP system right now with large numbers of computed indices, append-only OLTP inserts, and random OLAP reads; where the reads hold hundreds of locks each due to partitioning. Any time a write tx stays open for more than a few hundred milliseconds, the whole system degrades due to read locks piling up to the point that the DB begins to choke just on allocating and synchronizing them. The DB is only a few TB large, and the instance it's on has 1TB of memory and as many cores as one can get... but locking is locking.)

I wrote a blog about this for anyone who would like to learn more.

The query strings get you around the paywall if it comes up:

https://freedomben.medium.com/the-rules-of-clean-and-mostly-...

If anyone doesn't know what migrations are:

https://freedomben.medium.com/what-are-database-migrations-5...

My impression is that once you're at Facebook scale, most of your migrations are massive undertakings that need to take into account things like "How many terabytes more space do we need?", "How do we control the load on our DB nodes while the migration is going on?", "How do we get data from cluster A to cluster B?", "Is adding this index going to take hours and break everything?", and so on. Some of the time you'll be spinning up an entirely new cluster rather than changing the schema of an old one, and when you do migrate an existing cluster there's some five-page document specifying a week-long plan with different phases.

Then internally, they work around the inflexible db schemas by using offline batch processing tools or generic systems that can handle arbitrary data, for tasks that would be handled by the one DB in smaller systems.

That is largely the case.

For other, more complex cases where that is not possible, you migrate a portion of the userbase to a new db schema and codepath at the same time.