1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations.
Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails.
This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic.
There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)
Github Actions is CI/CD and it can be used to kick off any deploy step and manage it, but it's not one to actually perform the migration. That would be like Github Actions actually running the webserver directly.
In this example Github Actions needs to kick off a `eb deploy`, which it can totally do (I personally use CircleCI) but `eb deploy` (elastic beanstalk in general) is not designed to run a short lived script on a single container AND wait for it to return.
You can run a short lived script no problem, but you have no idea if it was successful or not because once the container becomes 'healthy' the command completes.
What I'm really waiting for is the container.lastStatus:'STOPPED' and exitCode: 0. Can't do that on EB.
I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not.
Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes.
That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result.
Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =(
Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it).
I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate.
I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.