Re:Emerging Architectures for Modern Data Infrastructure
by gfodor on Sunday October 18, 2020 @01:17AM
(#24815688)
Ah that makes sense. Thanks.
Re:Emerging Architectures for Modern Data Infrastructure
by fscherer on Sunday October 18, 2020 @07:41AM
(#24817127)
why is it actually that SQL "re-emerged" as the transformation layer? I thought that it first shifted from SQL to Query Builders inside talend, matillion etc. Why now SQL again?
What’s changed in the last 10 years are the access patterns. There’s increased demand to have arbitrary query access over the raw data. The most impactful technology changes have been about pushing the access layer (queries, stream & batch processing, dashboards, BI tools, etc) down as close to the raw data as possible and making that performant. What’s fallen out of that are better MPP OLAP databases (snowflake), new columnar formats (parquet), SQL as the transform layer (dbt).