News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by alexott on Monday January 30, 2023 @04:17PM (#34586499)
Even for data transformation logic, SQL isn’t the best choice. How would you handle the case when you need to apply the same transformations to few dozens or hundreds columns?
Re:SQL should be your default choice for data engineering pi... by thesz on Monday January 30, 2023 @04:37PM (#34586783)
Some SQL engines support generating and evaluating queries. I stumbled upon a function in MariaDB code base that is introduced and keps specifically for that use case (it returns properly quoted SQL value as a string, including "NULL" string for null values). It is aptly named QUOTE [1].

[1] https://mariadb.com/kb/en/quote/

You can then specify columns as a query, group_concat the result and evaluate resulting statement [2].

https://mariadb.com/kb/en/execute-immediate/

I hope that helps.

Re:SQL should be your default choice for data engineering pi... by cced on Monday January 30, 2023 @04:30PM (#34586689)
I'm not sure I understand the question, you can create tables/views based on others that have the transformations applied to them, tools like dbt[1] make this easy.

[1]: https://www.getdbt.com/