Re:The Evolution of the Data Engineer Role
by MrPowers on Monday October 24, 2022 @11:22AM
(#33317900)
Data warehouse: bundles compute & storage and comes at a comparatively high price point. Great option for certain workflows. Not as great for scaling & non-SQL workflows.
Data lake: typically refers to Parquet files / CSV files in some storage system (cloud or HDFS). Data lakes are better for non-SQL workflows compared to data warehouses, but have a number of disadvantages.
Lakehouse storage formats: Based on OSS files and solve a number of data lake limitations. Options are Delta Lake, Iceberg, and Hudi. Lakehouse storage formats offer a ton of advantages and basically no downsides compared to Parquet tables for example.
Re:The Evolution of the Data Engineer Role
by fabatka on Monday October 24, 2022 @11:55AM
(#33318360)
Can you explain why do you find the above explanation amusing? I honestly don't see the absurdity of it, although, my livelihood may depend on me not seeing it :)
Data lake: typically refers to Parquet files / CSV files in some storage system (cloud or HDFS). Data lakes are better for non-SQL workflows compared to data warehouses, but have a number of disadvantages.
Lakehouse storage formats: Based on OSS files and solve a number of data lake limitations. Options are Delta Lake, Iceberg, and Hudi. Lakehouse storage formats offer a ton of advantages and basically no downsides compared to Parquet tables for example.
Lakehouse architecture: An architectural paradigm to store data in a way such that's it's easily accessible for SQL-based and non-SQL-based workflows, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
There are a variety of tradeoffs to be weighed when selecting the optimal solution for your particular needs.