News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by hellodanylo on Monday January 30, 2023 @10:25AM (#34580863)
I find SQL very hard to use when the data schema and/or transformation graph needs to be dynamic (e.g. depends on the data itself). It's hard to make SQL dynamic even at build time -- Jinja+SQL is one of the worst development experiences I have ever had.
Re:SQL should be your default choice for data engineering pi... by bob1029 on Monday January 30, 2023 @10:33AM (#34580980)
> I find SQL very hard to use when the data schema and/or transformation graph needs to be dynamic

If the schema is "dynamic" then I'd accuse the business of being poorly-defined and not worthy of any development time.

Re:SQL should be your default choice for data engineering pi... by gilbert_vanova on Monday January 30, 2023 @10:40AM (#34581086)
That's great except for when you're interacting with a decrepit data system from 10 years ago with a variable record format.

Some things can't be locked in stone, and SQL will leave you out to dry when that's the case.

Re:SQL should be your default choice for data engineering pi... by lolive on Monday January 30, 2023 @10:44AM (#34581158)
I use graph database, and resources of the graph are typed with types/supertypes. Relationships also are typed with types/supertypes. And my queries are heavily dependant on that typing structure.

Honestly, I cannot live without that feature. [sorry, that is my OOP minute. Continue without me...]