Re:Engineers Shouldn’t Write ETL
by itronitron on Sunday November 04, 2018 @08:35PM
(#18379160)
I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.
I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.
They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.
Re:Engineers Shouldn’t Write ETL
by internet555 on Wednesday November 07, 2018 @08:09AM
(#18398841)
Really? I had one sort of similar because I took as many cross listed math electives as possible and avoided any software type courses, but even then I learned a lot less analysis, algebra, geometry than a good math student would learn
There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.