News for hackers. Stuff that matters.
Re:Systems design explains the world: volume 1 by glogla on Sunday December 27, 2020 @06:07PM (#25555040)
We just deployed rewrite of Hadoop (sqoop, hive, ...)/Teradata based Data Lake & Data Warehouse to Cloud based Serverless Lakehouse with Spark as the main workhorse.

This included migrating about 1/2 petabyte of old data and rebuilding several Kimball-style Data Marts to Data Vault approach, which pairs really well with the Lakehouse-style tech stack.

This all took about 9 months, which is a long time, but it was in highly regulated industry where compliance, security and data integrity get a lot of scrutiny.

In the end, nothing of the previous solution survived - no code and no infrastructure.

There are no comments on this comment yet.