To be fair, for 90% of business problems that require ML, I’d rather take the guy who throws XGBOOST at everything instead of the one trying to be fancy with neural networks. You get an explainable output and good results without deep subject matter expertise that the person likely won’t have. It also runs at a fraction of the cost.
To be fair though, most value from data science is driven by the analysis, not the models.
Is this different than your average SWE?
I've been a DS for 10+ years, and I feel the exact opposite. The worst "Data Scientists" I've worked with are all ex Software Engineers who seem to assume that business problems are really computation problems. So they find convenient ways to ignore the human aspects (e.g. trying to figure out why the data is a mess) and gravitate to using more complex algorithms and breaking down the problem to an achievable programming pipeline that runs in production, but the results are of low value. But it looks awesome on a resume.
Are you right or am I right about SWEs turned DS? I have no idea. But one quality that IMHO is important is the interest in actually looking at data and asking questions, which is much rarer than most people realize.
It doesn't sound much like your worst and their best is the same kind of person. I don't see necessarily conflicting views.
@OP - mind rerunning this analysis for "AI Engineer" titles? https://www.latent.space/p/ai-engineer anecdotally i saw 8 of these in the last Who's Hiring and wanted to tease out the emerging difference between ML and AI Engineer
So for newly minted Math PhDs, sure go out and learn how to do some ML coding in notebooks, but if you can't get a decent dataset together to train it on it'll be all for nought. Anyone with AI/ML coding only, and no SQL, is a no hire in my book.
The largest group of DS was non-ML/CS/Math PhDs who started panicking once they realized their future job prospects in academia were very slim and so they signed up for bootcamps and got jobs at places hiring DS by the hundreds. Many of the people in this latter group had no idea how to write Python outside of a notebook, generally just structured problems to fit into XGBoost, and when not doing that tried to squeeze resume-boosting-complexity into any problem the could find. They also tended to have a hilariously poor understanding of creating business value.
Nearly everyone I know in the first group has switched back to just being an engineer of some sort, typically ML or AI engineer. I suspect the small set of talented people from the second group will end up in lesser paying product analytics type roles or closer to product management roles, while the majority that don't bring much to the table other than a PhD will be slowly attritioned out of the field as companies start looking for the value different skillsets bring to the table.