The biggest error I see people do is coupling their DB migrations to their code deployments. Easiest way of handling this is to make each code version compatible with the DB schema before and after, and clean up the code after the migration was completed. Then you don't really care if the DB migration takes days to complete, nor if there are errors in the migration (unless you loose data obviously, then you're fucked). If the migration was wrong somehow, you can easily rollback the code as well and everything should still work, no need to rollback the migration just yet.
So most people seem to do migrations this way:
- Write migration file, commit to SCM
- When deploying the project, automatically run migration before starting application
- Wait for migration to finish, deploy code
What you could do to avoid issues like you mentioned:
- Write migration file to separate project
- Write code that works both with the version before applying the migration, and after
- Deploy new application code
- Apply migration when it suits you, application shouldn't care
- When confirmed it's working, clean up the code and deploy it again
This is only about the only way you can handle migrations that touches a lot of data and needs days to complete. But if you haven't reached that scale yet, your migrations are probably still coupled to your code deployments, which is probably fine in most scenarios, but can always be better :)
That tends to be the norm with Terraform, too :)
If it's like Terraform, you still need to know exactly what the tool is doing and how the services behave under the hood.
So much for Ops setting expectations.
What happens when it gets 53% of the way through whatever it's doing at it errors out? Now wtf do I do?