Re:Databricks open-sources Delta Lake to make data lakes mor...
by cavisne on Wednesday April 24, 2019 @11:44PM
(#19744786)
The number of definitions here suggest its not well defined, one more :)
A data lake is when you store your data in object storage (S3, GCS etc) as opposed to a filesystem (HDFS) or some indexed datastore (Redshift etc).
This potentially saves a lot of money because you can scale compute separately from storage, and object storage can be extremely cost effective compared to running a distributed filesystem.
Where the two overlap is when you store something like parquet in object storage, the file format is somewhat indexed already so you spend a bit more money preprocessing it but save a lot of money querying it.
I think whether its "raw" json or log files or preprocessed parquet doesnt really differentiate whether its a data lake or not
A data lake is when you store your data in object storage (S3, GCS etc) as opposed to a filesystem (HDFS) or some indexed datastore (Redshift etc).
This potentially saves a lot of money because you can scale compute separately from storage, and object storage can be extremely cost effective compared to running a distributed filesystem.
Where the two overlap is when you store something like parquet in object storage, the file format is somewhat indexed already so you spend a bit more money preprocessing it but save a lot of money querying it.
I think whether its "raw" json or log files or preprocessed parquet doesnt really differentiate whether its a data lake or not