At my job the DB is backwards compatible by one release. This means we roll back code, not data. It takes a little longer to do something like delete a column, but it's worth it.

Rainforest QA had a blog post on strategies a while ago.

Simply testing that you maintain backwards compatibility is a royal pain though.

One could imagine a test rig which created a db with test users in it, then ran v1 of the software, then v2, then v1 again, and checked v1 didn't crash on the now mutated datastructures.

Most datastore mutations might only happen on certain user actions and inputs, so making sure you get full coverage would be rather tricky.

They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...

From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second.

I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.

At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal.

Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.

I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.

Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.