An IT migration corrupted 1.3B customer records

Re:An IT migration corrupted 1.3B customer records by jazzkingrt on Tuesday December 24, 2019 @09:04AM (#21872101)

What I've seen work well, for multiple migrations, (at an admittedly smaller scale) is using shadow writes/reads and a source-of-truth toggle.

The API layer made requests to both the old DB and a new DB that had been populated during a small window of scheduled downtime.

We spent a couple weeks/months running checks in production that the old DB and new DB were returning identical results, though still returning the old DB's results as source of truth. Eventually, we flipped the source of truth to the new DB, and some time later decomissioned the old DB.

[ Parent | on HN ]

Re:An IT migration corrupted 1.3B customer records by Mouse47 on Tuesday December 24, 2019 @10:28PM (#21876576)

Great approach imo. Once you get flipped over to the new source of truth you have to make sure the business prioritizes decommissioning the old DB, though. I've seen certain departments (looking at you, BI) treat the old database as a permanent backwards compatibility layer.

[ Parent | on HN ]

Re:An IT migration corrupted 1.3B customer records by 8note on Wednesday December 25, 2019 @02:47AM (#21877466)

when the old database stops getting writes, it stops being useful pretty quickly

[ Parent | on HN ]