News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by onlyrealcuzzo on Monday January 30, 2023 @09:30AM (#34579980)
> Manipulating the data? Sure, use SQL, as long as you're operating on data from one database. Trying to collate data from multiple databases may have unpredictable performance issues even in the engines that support it.

Depending on how big your data & org is - orgs solve this by writing their own query engines that can query many different data formats.

Re:SQL should be your default choice for data engineering pi... by crest on Monday January 30, 2023 @12:25PM (#34583132)
And why shouldn't this engine accept SQL queries as well if it's accessing existing SQL databases (probably just with different weights, retries, and batching/spooling)?
Re:SQL should be your default choice for data engineering pi... by onlyrealcuzzo on Monday January 30, 2023 @12:53PM (#34583621)
It usually does!

If you want to use the same logic / language to query some exotic dataset for specific use cases - it can often be worth it to write a custom engine that can do it - rather than expect all end-users to learn the ins & outs of the other databases & datasets (not to mention, to learn something beside SQL).

Instead, a single team (the query engine owners) can optimize the query engine - rather than individual users trying to optimize every script individually.

Your users can become masters of their engine / language (SQL) - because it can be used for the vast majority of cases.

For many reasons - you might want to store data in a format that MySQL / Postgres does not support natively (see Google Spanner). But, ideally, you'd still be able to leverage the fact that almost every programmer in the world can write SQL.