There’s a lot of truth in this, but I find the people that pivot toward practical software engineering skills early in their PhD are still highly valuable to tech companies.

I think the issue here is you can’t become highly skilled in software engineering overnight, and compensating by flexing softer “problem solving skills” isn’t enough to compensate. If you want to be effective in data science, you need to commit to being a rockstar programmer AND statistician AND be well versed in a number of different algorithms.

Unsurprisingly, this is difficult, and so the market of people who do it well is small. That’s not, however, a referendum on the value of predictive modeling and software engineering. I don’t see advanced math and software engineering skills going out of fashion anytime soon, simply because they are hard to acquire and immensely powerful in the right hands.

I wonder what the impact of easy-to-use cloud ML tools is going to be on the industry. BigQuery already lets users train models using a simple SQL query. AutoML makes simple ML tasks as easy as run and verify accuracy and run again. Of course, a basic amount of knowledge is still need to avoid common pitfalls - unclean data, overfitting, etc. Perhaps this will just mean data scientists can produce better work with the same amount of time. It doesn't seem like a good use of everyone's time to have to know so much cutting-edge research to solve run-of-the-mill business problems. Hardly any other area of CS requires that.

I would say, however, there is ample opportunity for infrastructure designers/engineers in the space, especially in a world where everyone is going "cloud" but cloud doesn't cut it in the most demanding applications, so those of us who love data center design are sort of loving the resurgence of onprem/Colo usage and the opportunities for pushing the envelop on data transfer/processing rates, etc. The skills and knowledge gained are very useful in other industries, with a side bonus of, if you pay attention, knowing a bit more about the "big picture" of AI than many of the phd's!

For readers who read that and think they still want to do it:

> The vision of AI/ML/DS has turned into a nightmare of people shoving data they don't understand into models they don't understand...

Don't be this ^. Please.

If you do want to come into the field, please first take the time to understand the tools. Learn enough to re-derive and re-implement the tools you're using. Learn the assumptions that need to hold. Don't just learn to point the tools at data and hit go. It's not like a lot of programming where it either succeeds or fails. Failures are often silent in this field. If you can't reason through how and when a given tool will succeed and, probably more importantly, reason through how and when a given tool will fail, you haven't learned enough to know what to use and when.

I totally agree with you, but let's be honest, in this publish-or-perish world in academy, and the cut-throat environment of VC-funded start-ups, people will, if not outright lie, exaggerate the nature and importance of what they are doing. Too much money and "prestige" is at stake.

The "right" way is slow, lonely and unglamorous. And most importantly without that fast money, acquisitions and promotions.

> The explosion of AI/ML/Data Science teams in places that they really don't belong is going to have major backlash soon

I'm not disagreeing with you, just curious, did that happen with Big Data teams? If not, what happened with all the people working around that concept a few years ago? Might be an indicator of what could happen to DS teams.

I have heard hiring managers explicitly saying that data engineer role on your resume is considered a negative point for them.

Every time i fire a data scientist the earnings grow