I very much disagree. If your bad deploy included migrations that can't be reversed (drop an old table for example) then rolling back just gave you two problems.

As long as the dev wrote (and tested!) two-way migrations and they are possible, then yes you are correct.

The long-term viable approach is to not make forward-backward incompatible migrations in short period of time.

If your system is truly distributed (multiple machines hosting it), then if one server performs migration that deletes the table, then the other server stops working.

You must have 2 checkins and 2 rollouts: Create a new table while maintaingin the old one -- and let it bake in; and then delete the old table a week later (or whatever is your cadence/release cycle).

IMHO the answer here is to just never do irreversible migrations. (In other words: just leave old tables/columns around--if not indefinitely, then for a few weeks after they stop being used.)

I have done that in the past and I agree, it's a decent solution. The risk is that the old stuff never gets cleaned up and before long your DB is full of all kinds of cruft and nobody knows which part is needed and which isn't.

I saw a heinous bug once because a newer person was using a column that had stopped getting updated years earlier (because it was replaced). This person saw the column and it was exactly what they needed. Then customers started getting billed on old accounts that they had either closed or changed with us. They were really mad.

My solution to this is to rename that table/column to “table_deprecated” or “column_deprecated”. This has the nice property of being reversible, causes nice visible errors if something unanticipated is actually using the column/table, makes it obvious that it shouldn’t be used (well, one would hope), and makes it easy to find for permanent deletion later.

That's a great idea! Although I did work with a person once who would use anything that looked helpful regardless of "deprecated" being all over the place (Eclipse would even yell at the method calls but he didn't care). Granted that's Java and at that point in time APIs were being deprecated without workable replacements, so it's hard to fault him.

At that point you might as well just delete them.

I think you're missing the point. If your migration renames a table or column, then the migration can be rolled back, by simply reversing the rename. If you delete the thing, then there is no way of rolling back other than restoring from backup.

I can see it being useful to rename these things, as long as there is some process in place to delete the renamed items after a period of time has passed.

Would be good if you could schedule a future migration that will happen only after a certain date has passed, which deletes the renamed tables/columns.

This sounds so much like something that I'd do, that I had to stop and think about my personal hall-of-fame screwups to make sure that wasn't one of them.

In terms of things that scare me as a developer, stale data is WAAAAYYY less concerning then having people try to shotgun fix an issue on a production server after an irreversible update took down the entire system... Especially now that the developer and the entire organization is likely now in a panic state and not thinking clearly.

That bug you describe, maybe stale data is part of the problem, but ... why are you having a new person working on the billing system presumably without any code review? It sounds like you have a lot of process issues.

Just create an issue in your issue tracker with the highest priority to delete a column (and why) with a deadline in 4 weeks. Stupid simple solution but works perfectly.

"Clearly the answer for any software problem is to just not have any bugs."

You see how that isn't an actual solution?

An irreversible migration isn't a bug, nor is anyone saying the answer for any software problem is to not have bugs.

They're saying the solution to the specific problem of being unable to rollback due to irreversible migrations is to not write irreversible migrations, which is a completely valid solution and indeed the correct one. The whole point of migrations is to track db changes so that undoing them is easy.

That misses the point.

Code and the DB always need to be compatible one version forward and back. That's required engineering discipline.

creating a table is an irreversible migration. Once you have the data, you can't just ... reverse the migration by deleting the whole table.

I would never ever EVER want to do a deploy with migrations that can't be rolled back. That sounds like professional malpractice TBH.

Two way migrations are both possible and SHOULD be done for any real data.

> Properly designed container infrastructure takes the guesswork and stress out of deploying. Period. Fact. If yours doesn't, it's not set up correctly.

Hmmm, I don't have the experience to know if it's setup correctly or not. All I can do is watch it fail and then learn from my mistakes.

Is there a container "framework" that out of the box gives me all of " Blue/Green, canary, auto deploys, rollbacks..." so I don't have to guess if I'm doing it right?

I hate to say it, but yeah that's kind of the point of kubernetes deployments (https://kubernetes.io/docs/concepts/workloads/controllers/de...). Or openshift for more UI and "out of the box" experience.

You deal with all the headache of making your app stateless with a predictable API so that you can reap the benefits of a system like k8s, which can automatically manage all of it for you.

Similarly i'm a bit confused by your comment about SSH dying... in k8s you configure a readiness/liveness probe and behavior when the probe starts to fail. If SSH is an important thing for a given container, maybe the "liveness" probe is the command "ps aux |grep sshd". Then if it dies, the container can be pruned automatically.

Nomad [1] does it as well, also visualized nicely in their awesome UI.

[1] https://learn.hashicorp.com/tutorials/nomad/job-blue-green-a...

We’ve been using Convox[0] for the last 2 years. I’ve been pretty happy with how simple it is to work with. We’re still on version 2 which uses AWS ECS or Fargate. Version 3 has migrated to k8s and is provider agnostic. We just haven’t had the bandwidth to upgrade yet.

[0] https://convox.com/

We are using Convox v2 too and are happy with it, but I'm hesitant to do the upgrade to introduce the complexity of kubernetes to our devs and if convox the right abstraction on top of kubernetes when there which is already a pile of abstractions in k8s itself (and so many other tools to choose from in the k8s universe).

https://github.com/aws/copilot-cli isn't ready for our use cases, but is more or less convox v2 built by AWS.

I remember that... But it was too aws specific. I'll be giving it another look, thank you for pointing that out!

Edit: sadly it's still another layer onto aws/gcp/other... I'll pass again.

This seems overly absolute to me. What about all of the cases where the bug wasn't caused by a recent commit? Some cases of this I've seen are:

* Time bomb bugs. Code gets committed that works fine until some future condition happens, such as a particular subset of dates that aren't handled properly.

* Efficiency issues. Some code could might function properly and work fine with low amounts of data, but hit a wall when it has to handle loads beyond a particular size.

* Bugs in code that just hadn't received much traffic yet. A feature having a bug that only affects 0.1% of people using it might not be discovered until the feature gains traction down the line.

Having dealt with all of these in production, I can tell you the strategies I've used to combat these things:

1. Solid code reviews. Anyone of our developers can halt a code review for any reason. We require 3 approvers on each review. Sensitive areas require reviews from people familiar in that area. We also have tooling that allows us to generate amounts of test data in dev that is similar to prod loads. This helps us catch a lot of time bombs.

2. Feature toggles to decouple deploy of code from release of code. This allows us to test our code in production before turning it on for customers. It also allows us to slowly rollout a feature and watch how the code behaves. This also gives us a kill switch to turn off the code if it is bad.

3. An incredibly robust testing pipeline. It takes about 50 minutes from commit to production deployment. We can also deploy previous containers very quickly for situations that require it.

This doesn't solve all of our problems. Some changes cannot go behind feature toggles (DB migrations, dependency upgrades, etc). But we do pay a lot of attention to design and rollout plans for database migration changes and such.

All of these things come at an extra cost to us, but it allows us to move quickly when we need to. But we're in a lot better place than we were when we were trying to do weekly releases. We have a good mix of team experience (sr vs jr) - and have a lot of discipline in our software engineering practices. We still have problems like I said, but these strategies have greatly improved our ability to deliver software.

Out of curiosity, how many devs does your org have? I think a lot of the disagreements here come from people at orgs with 3 developers talking to people at orgs with 30,000.

about 40 and growing

Sometimes you have to quickly roll forward.

Those sometimes should be super rare, and you should build testing infrastructure to prevent that from needing to happen. When you release something, you move traffic from the alb over to the new instance, if you have an issue, just move it back. If you are deploying breaking changes and don't provide yourself a stable upgrade and downgrade path, yea, you're gonna have trouble.

I’ve learned to take the time and go through the normal deploy steps for any hot fix. More often then not, rushing the steps leads to longer outages, missing the actual bug, creating a new bug, etc.

Don’t cowboy it, deploy properly and you’ll be more relaxed in the long term.

Yeah, I should have been more clear. I'm 100% for using normal deploy steps and I'm not recommending cowboy-ing updates in a container.

He was asking about using non-containerized infr though. If you can commit a code hotfix and quickly deploy the code package, you can roll forward without the slow container build/deploy.

Incidents _requiring_ rolling forward are extremely rare. In the cases you have to, just build the image and deploy to your cluster with a high max-surge configured.

If you image has correct caching, rebuilding it shouldn't take much time. Most of your time is likely spent in CI and rolling deployments, both of which you can manually skip.

> If you image has correct caching

This is the hangup for most CI/CD systems with containers. Typical configurations (e.g. Gitlab basic setup) don't leverage any caching, so every container is built 100% from scratch, every time.

Adjusting the system to properly utilize caching and ordering your container builds in a way that the most volatile steps are as late as possible in the build will massively speed up container builds.

Please do not.

My guess is that they are writing code (migrations...) Without thought given to rollback.