Re:Databricks open-sources Delta Lake to make data lakes mor...
by mucio on Thursday April 25, 2019 @03:02AM
(#19745699)
To storage and compute I would also add metadata, this is why most data lake projects end up in data swamps.
When people just dump data in their storage they end having really hard time sharing them in their organization.
So amongst the cloud providers, AWS calls a combination of S3 + Glue + Athena (for example) a "data lake", where S3 is the object storage which can store data in various formats, and Glue and Athena are used to transform/process/query the data. See a more detailed article/guide here: https://aws.amazon.com/lake-formation/
If you didn't want to put anything into the cloud and keep all your services on-premise, a local Hadoop cluster could be a data lake, for example using HDFS + Zookeeper + YARN + Hive.
[This is a huge over-smiplification because it's late and I really should be going home :)]