News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by crest on Monday January 30, 2023 @12:25PM (#34583132)
And why shouldn't this engine accept SQL queries as well if it's accessing existing SQL databases (probably just with different weights, retries, and batching/spooling)?
Re:SQL should be your default choice for data engineering pi... by onlyrealcuzzo on Monday January 30, 2023 @12:53PM (#34583621)
It usually does!

If you want to use the same logic / language to query some exotic dataset for specific use cases - it can often be worth it to write a custom engine that can do it - rather than expect all end-users to learn the ins & outs of the other databases & datasets (not to mention, to learn something beside SQL).

Instead, a single team (the query engine owners) can optimize the query engine - rather than individual users trying to optimize every script individually.

Your users can become masters of their engine / language (SQL) - because it can be used for the vast majority of cases.

For many reasons - you might want to store data in a format that MySQL / Postgres does not support natively (see Google Spanner). But, ideally, you'd still be able to leverage the fact that almost every programmer in the world can write SQL.