Re:SQL should be your default choice for data engineering pi...
by ramraj07 on Monday January 30, 2023 @09:56AM
(#34580339)
Do you mean you ask them to edit the sql?
With some oversight this can be fine but non engineers can easily end up making the sql infinitely slower not to mention get things wrong (most commonly doing inner joins or have nulls in where in joins).
Re:SQL should be your default choice for data engineering pi...
by tracker1 on Monday January 30, 2023 @11:32AM
(#34582126)
I've seen plenty of devs and da's do the same. The nice thing about SQL is it's easy enough to create a query that gives you what you want, but at scale it falls flat or purforms poorly. It's easy enough to work through most of the time, but too many lack the understanding of knowing when bottlenecks are likely to happen, and if/when it may be an issue.
I think of more than a couple basic joins in a query to be a code smell.
Re:SQL should be your default choice for data engineering pi...
by bob1029 on Monday January 30, 2023 @10:49AM
(#34581258)
We do the same thing with our business logic & product configuration. I still haven't found something I couldn't expose as a SQL configuration opportunity.
Even complex things where you have to evaluate a rule for multiple entities can be covered with some clever functions/properties.
I've had a ton of success asking our marketing and business teams to own changes and updates to their data - very often they know the data far better than any engineer would, and likewise they actually prefer to own the business logic and to be able to change it faster than engineering cycles would otherwise allow.