> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic.

What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this:

Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Their work is less creative. When they are successful, their work is mostly invisible. They are interchangeable. People think of what blacksmiths do as more like scripting than writing code. Blacksmiths mostly work on configuring systems they didn't build. Blacksmiths do more troubleshooting than building. Their roles are focused on support.

Data scientists ("magicians"): Magicians are paid more. Much more. People think of them as more highly educated. By definition, what they do is magic. They work on prominent projects. Their successes are highly visible. They build large systems that only they can comprehend. They use support staff to clear away mundane obstacles so they can focus on unique, highly creative aspects of work.

Saying that we need more data engineers than data scientists is like saying that we need more janitors than CEOs. That's true, but it's true because we made it true by structuring projects around one prominent, well-paid person supported by a staff of invisible drudges.

This smacks of the positive self-talk that QA and software testers used to give each other: "We are indispensable! We take pride in our craft! Nothing can ship without our signoff!" And then lots of companies reduced their QA or eliminated it wholesale by focusing on continuous delivery and changing consumer expectations of what "broken" or "acceptable" means. The same fate awaits data engineers.

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers.

Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?

If something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.

I remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.

The way the evolution in software went, platforms became more capable and allowed individuals to automate more common tasks. QA/DevOps/SRE teams were consolidated and replaced with smaller platform teams which empowered internal engineers to quickly write scalable and well tested services. If data management, instrumentation, and ML tooling become sufficient then perhaps the data engineers will be replaced by a science platform team.

Caveat is that many scientists are expected to publish novel research to advance their career. "Infrastructure" and "data management" do not tend to produce the kinds of sexy projects which are attractive to publish.

I think this model (an integrated team) is what I see, there is a huge benefit in terms of short decision loops from having one team - but data engineering skills are really important in enabling it. Also if the people doing the data engineering are close to the data science then there's much more likelihood that they will produce effective solutions for the backend of the project.

aye - this pattern often trends towards the roles merging over time. The counter example to the platform team approach from the software side is Software Engineers owning the full infra and ops that their services need.

There's a big incentive for companies that need to hire more people to give the "better" title out for the same kind of work. If managers do maintain a gold/silver role on their team all of the folks in the silver role will look at the gold role as their next move. Worse there is a net-negative productivity drag where the gold/silver role constantly debate what's in-scope vs. out of scope for their work.

I once saw a team where the scientists were meant to be equivalent to SDEs in coding skill, but the scientists could only in practice do some light python/bash scripting. They tried to make the SDEs responsible for "productionalizing" the projects which meant adding tests/etc. The engineers who could all left the team in 6 months, the ones who remained were also unable to perform more than light bash scripting/python work.

>reduced their QA or eliminated it wholesale

This is usually the case only for companies that work on low risk applications (I.e. not safety related or critical industries) or have been lulled into complacency (sometimes, ironically, “we haven’t had a major issue so obviously QA isn’t needed” when strong QA is precisely why they didn’t see issues)

  > low risk applications (I.e. not safety related or critical industries)

My anecdotal experience is with Palantir (software used in war zones). Between 2016-2018, they eliminated most of the 150-200 QA people they had. Testing is now done by devs and users with the help of sound CD principles like blue/green deployments.

That’s fair, because there’s process elements in place to mitigate risk. I read your comment to say “get rid of QA” not “get rid of the separate QA team”. My comment was more pointed to those who feel testing is inherently wasteful. In your example, the testing and configuration is tightly controlled to mitigate the same risk as a QA than (although there may be something to be said about the best practice of having the QA team be independent). Where I get nervous is when testing gets cut in hazardous systems because of cost or schedule. I personally wouldn’t want to get on a plane or autonomous car built like that. My own personal anecdotal experience is that organizations that were cavalier about QA on safety critical hardware/software inevitably had their comeuppance

>Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting)

>management may consult the blacksmith in the selection of magicians

I mean when you put it like that, why hire a magician (bullshitter), if the magic relies on the blacksmith?

And, if management needs to consult someone (a blacksmith) on hiring (another blacksmith or for whatever reason a magician), then arguably management is made of magicians.

Don't get me wrong, I agree with the point your making. It's just the problem is with BS and BS is rampant, or like you say: a dime-a-dozen.

> I mean when you put it like that, why hire a magician (bullshitter), if the magic relies on the blacksmith?

Because magicians are better at the smoke and mirrors that drives funding rounds and closing big sales deals.