Yes that's how you'd do it. That said, while I really love SQLAlchemy and worked with Alembic for a long time I switched to manually generated SQL-based migrations in all of my projects, as there are simply too many corner-cases that are quite hard to capture with tools like Alembic. Writing plain SQL files gives you much more expressive power and also makes it quite easy to build complex migrations that e.g. involve first some schema changes, then a data migration, and then some more schema changes, which is hard to do with most migration tools.

I also found that the use-case which most of these migration tools optimize for (being somewhat independent of specific SQL dialects) rarely works in practice, except maybe if you stick to the smallest common denominator of all SQL dialects. And that again is wasteful IMO as not using a lot of the functionality that's present e.g. in Postgres can make your life a lot harder.