Re:Online migrations at scale
by mighty_warrior on Friday February 03, 2017 @10:51AM
(#13560602)
They maintained a replication process across both tables as they updated the read processes before updating the write process. Say for whatever reason their offline replication process broke for 2 hours. For those 2 hours of downtime that replication is broken, the system is reading from a table that is not in sync with the table that is receiving writes. At that point you are displaying incorrect subscription data to your customers.
> They maintained a replication process across both tables as they updated the read processes before updating the write process. Say for whatever reason their offline replication process broke for 2 hours.
From the article I got the impression that both tables were being written to in the same database transaction, so this is not a possible failure scenario at all.
Such great pains come with huge systems. What's the alternative?
Taking the platform offline for a few hours? Management will say no. Or maybe Management will say yes once every three years, severely limiting your ability to refactor.
Doing a quick copy, and hope nobody complains about inconsistencies? Their reputation would suffer severely.