News for hackers. Stuff that matters.
Re:We don’t use a staging environment by kgeist on Sunday April 03, 2022 @04:48PM (#30900494)
>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version

Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev environment but in production it would lead to timeouts in requests for the duration of the migration for large clients (our application is sharded per tenant) because the table was very large for some of them and the migration locked it. The staging environment helped us find the problem before hitting production because we routinely clone production data (deanonymized) of the largest tenants to find problems like this. It's not practical (and maybe not very legal too) to force every developer have an up-to-date copy of that database on every VM/laptop, and load tests in an environment very similar to production show more meaningful results overall. And feature flags wouldn't help either because they only guard code. So far I'm unconvinced, it sounds pretty risky to me to go straight to prod.

I agree however that the concern about conflicts between feature toggles is largely a monolith problem, it's a communication problem when many teams make changes to the same codebase and are unaware of what the other teams are doing.

Re:We don’t use a staging environment by nicoburns on Sunday April 03, 2022 @05:04PM (#30900631)
> Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

This is definitely best practice, but it's not strictly necessary if a small amount of downtime is acceptable. We only have customers in one timezone and minimal traffic overnight, so we have quite a lot of leeway with this. Frankly even during business hours small amounts of downtime (e.g. 5 minutes) would be well tolerated: it's a lot better than most of the other services they are used to using anyway.

Re:We don’t use a staging environment by withinboredom on Sunday April 03, 2022 @06:34PM (#30901410)
> Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place.

This doesn't have to be true. You can create an entirely separate table with the new data. New code knows how to join on this table, old code doesn't and thus ignores the new data. It doesn't work for every kind of migration, but in my experience, it's preferred by some DBAs if you have billions and billions of rows.

Example: `select user_id, coalesce(new_col2, old_col2) as maybe_new_data, new_col3 as new_data from old_table left join new_table using user_id limit 1`

Re:We don’t use a staging environment by charrondev on Monday April 04, 2022 @08:00AM (#30905631)
For us we use pt-online-schema-change which copies the table first, sets up some triggers to keep things synced, then renames the tables at the end.
Re:We don’t use a staging environment by atmosx on Monday April 04, 2022 @01:16AM (#30903517)
> because we routinely clone production data (deanonymized)

Are you using an external service or in-house tool to perform this operation?