News for hackers. Stuff that matters.
Re:Ask HN: How do you do slow migration with zero downtime? by andrewfromx on Sunday April 02, 2023 @05:56AM (#35408906)
can you break up your migrations into smaller chunks that can be run incrementally?

There is also:

AWS Database Migration Service https://aws.amazon.com/dms/

Azure Database Migration Service https://azure.microsoft.com/en-au/products/database-migratio...

Liquibase CI/CD for Databases https://resources.liquibase.com/ci-cd-database

In PostgreSQL, you can use the "CONCURRENTLY" keyword with the "ALTER TABLE" statement to perform certain operations without locking the table and without requiring downtime.

ALTER TABLE table_name ADD INDEX CONCURRENTLY index_name (column_name);

DynamoDB offers a feature called "Online Indexing" that allows you to create or delete a global secondary index (GSI) without any downtime to your application. This feature allows you to add, remove, or modify an index on a table without any impact on read or write traffic to the table.

aws dynamodb update-table --table-name table_name --attribute-definitions AttributeName=attribute_name,AttributeType=S --global-secondary-index-updates '[{"Create":{"IndexName":"index_name","KeySchema":[{"AttributeName":"attribute_name","KeyType":"HASH"}],"Projection":{"ProjectionType":"ALL"},"ProvisionedThroughput":{"ReadCapacityUnits":5,"WriteCapacityUnits":5}}}]' --online-index-upgrade True

aws dynamodb update-table --table-name table_name --global-secondary-index-updates '[{"Delete":{"IndexName":"index_name"}}]' --online-index-upgrade True

Elasticsearch offers a feature called "Zero Downtime Indexing" that allows you to create, update or delete an index without any downtime or interruption to the search functionality of the index.

To create an index with zero downtime in Elasticsearch, you can use the "alias" feature

PUT /new_index { "aliases": { "my_search_alias": {} } }

After indexing data into the new index, you can switch the alias to the new index:

POST /_aliases { "actions": [ { "add": { "index": "new_index", "alias": "my_search_alias" } }, { "remove": { "index": "old_index", "alias": "my_search_alias" } } ] }

Re:Ask HN: How do you do slow migration with zero downtime? by edweis on Monday April 03, 2023 @09:03AM (#35422799)
Using the alias is very handy indeed. I think generally everything you mentioned and linked is the best practice. I am amazed that it still need to be implemented for each of us and there is no third-party to "fix and forget" data migration.