SQL should be your default choice for data engineering pipelines

Re:SQL should be your default choice for data engineering pi... by ramraj07 on Monday January 30, 2023 @09:56AM (#34580339)

Do you mean you ask them to edit the sql?

With some oversight this can be fine but non engineers can easily end up making the sql infinitely slower not to mention get things wrong (most commonly doing inner joins or have nulls in where in joins).

[ Parent | on HN ]

Re:SQL should be your default choice for data engineering pi... by tracker1 on Monday January 30, 2023 @11:32AM (#34582126)

I've seen plenty of devs and da's do the same. The nice thing about SQL is it's easy enough to create a query that gives you what you want, but at scale it falls flat or purforms poorly. It's easy enough to work through most of the time, but too many lack the understanding of knowing when bottlenecks are likely to happen, and if/when it may be an issue.

I think of more than a couple basic joins in a query to be a code smell.

[ Parent | on HN ]