Re:Systems design explains the world: volume 1
by glogla on Sunday December 27, 2020 @06:07PM
(#25555040)
We just deployed rewrite of Hadoop (sqoop, hive, ...)/Teradata based Data Lake & Data Warehouse to Cloud based Serverless Lakehouse with Spark as the main workhorse.
This included migrating about 1/2 petabyte of old data and rebuilding several Kimball-style Data Marts to Data Vault approach, which pairs really well with the Lakehouse-style tech stack.
This all took about 9 months, which is a long time, but it was in highly regulated industry where compliance, security and data integrity get a lot of scrutiny.
In the end, nothing of the previous solution survived - no code and no infrastructure.
This included migrating about 1/2 petabyte of old data and rebuilding several Kimball-style Data Marts to Data Vault approach, which pairs really well with the Lakehouse-style tech stack.
This all took about 9 months, which is a long time, but it was in highly regulated industry where compliance, security and data integrity get a lot of scrutiny.
In the end, nothing of the previous solution survived - no code and no infrastructure.