Re:Single dependency stacks
by jaxrtech on Wednesday February 09, 2022 @03:42PM
(#30278205)
Absolutely. I've also seen my fair share of horrendous home-grown "ETL" programs that waste more time shuffling bytes to and from database with poor ORM queries in loops, that could be done with a couple half decent SQL queries.
Probably the most useful things for me was learning relational algebra in college, and having been thrown on the deep end on a team that was very SQL heavy (not withstanding attempting to debug Oracle PL/SQL syntax errors while pulling your hair out about a missing closing parenthesis -- of which isn't the problem).
The usual challenge seems to be fetching data from external services or performing complex business that may conditionally load things -- things that can be awkward in procedural SQL. At the end of the day, you're building a messy ad-hoc dependency graph that is being manually executed very inefficiently. Would be better to just have your code just describe the dependency graph and treat each value transparently as a promise/future and then have a separate engine execute it.
Anyhow, something something monads with lipstick, I digress...
Re:Single dependency stacks
by PeterisP on Thursday February 10, 2022 @11:12AM
(#30288261)
Actual ETL systems would not, however, if you have a homegrown pile of scripts fulfilling the ETL role, I would not be surprised if those scripts would use ORM.
Probably the most useful things for me was learning relational algebra in college, and having been thrown on the deep end on a team that was very SQL heavy (not withstanding attempting to debug Oracle PL/SQL syntax errors while pulling your hair out about a missing closing parenthesis -- of which isn't the problem).
The usual challenge seems to be fetching data from external services or performing complex business that may conditionally load things -- things that can be awkward in procedural SQL. At the end of the day, you're building a messy ad-hoc dependency graph that is being manually executed very inefficiently. Would be better to just have your code just describe the dependency graph and treat each value transparently as a promise/future and then have a separate engine execute it.
Anyhow, something something monads with lipstick, I digress...