As someone who dabbles in the field from a strong software engineering background but only MOOCs on deep learning, absolutely this. The courses are often so focused on getting you to reproduce the state of the art you are on rails the whole time with no deep intuition for why things work. And so when I get off the rails and on a real problem or a Kaggle I often find myself training terrible models and not knowing why, getting stuck on bad architectural decisions, or just forging ahead with a terrible dataset without realizing it.

I’m reminded of the “draw the rest of the owl” meme. Except these mistakes are more insidious, since everyone can tell if they have drawn an owl, but it can be hard to tell if your model has done something like overfit on minutia until you release it upon the world.

No worries, even PhDs don't understand why things work, only how ;-)

You simply need to practice on challenging problems, sometimes ripping off code that is state-of-art, analyzing it, understanding why certain things are done the way they are done, and in a few months you could be able to do your own state-of-art custom models with your own loss functions with distributed training using your own callbacks and getting $500k salary in some valley company.

> even PhDs don't understand why things work

When you finish your Bachelor's degree, you know everything.

When you finish your Master's degree, you realize you know nothing.

When you finish your PhD, you realize nobody else knows anything, either.

what IQ do you think is required for this? do you think a solid lead dev who knows linear algebra and calculus could do this in a year or so?

With my first job after college, I found a large impedance mismatch between my (mostly mathematical) education and engineering in the physical world. It took some time to overcome that, but in the end having the strong math basis turned out to be a big advantage.

(Better math meant designs could be optimized for lighter weight and reduced manufacturing cost.)

I think you need both for machine learning systems, and the software engineering one is easier to teach on the job than math education. Or at least we expect to have to grow candidates that way and have infrastructure for it.

Remember this was the height of MOOC madness.

Then we learned students never completed moocs and instructors hated creating content and getting paid almost nothing (compared to their full time jobs) to be a sharecropper on their platform, without tenure, status, or all the other perks teachers enjoy (because they don't get paid much). Contrary to techies opinions, teachers do not want to give up their hard won political power ( that's all they have left ) for the sake of tech.

They saw what tech did to the publishing industry

I can't help but think that both of those issues are eminently solvable. In the case of teachers, focus on making a product for the teachers, then charge them either a flat fee or a percentage, ala kickstarter or patreon.

As for the nobody finishes, that is even easier: use Beeminder[0], and if you are writting a mooc platform, integrate with it.

[0]: beeminder.com; self-control as a service; I am a very happy user, but, except for a few stickers, an unpaid shill.

If only we could prevent college dropouts with beeminder :)

> which is much harder than it was in 2016 due to increased competition, and the requirement for Masters/PhD nowadays

The requirement for most ML jobs in 2012 (when I graduated) was a Ph.D. I took literally every course on machine learning my university offered (six or so) as an undergrad with the hopes of landing an ML position when I graduated. I gave up on trying to find a position because they all required Ph.Ds.

I even had a solid portfolio, as I had successfully applied neural nets to solve a couple of problems by then. One involved real-time image processing and it was on github.

It seems much easier to get a job in the field now.

Why would you say it's easier now? Even now there's this requirement of PhD

The proliferation of libraries that make it trivial to do machine learning. I know plenty of professional data scientists with liberal arts/business degrees who never completed calculus.

I'd say the bulk of the data scientists I work with now are non-cs/math people. I work with some math/stats PhDs too, but they are outnumbered like 3 to 1.

What kind of "soft skills" do you mean? Typically when I hear "soft skills" I think "interpersonal communication, leadership, organization ability" type things.

I never really understood what showing your work meant in my schooling before college. I'd solve the problem through whatever I had memorized and would occasionally jot down the more complex parts like long division or large multiplications, at least until I was allowed to use a calculator. In high school we started getting to more complex things like algebra and inequalities and I was able to power through with intuition and a solid proficiency with my TI-83 and some Basic.

I don't know exactly what my teachers were doing but it all came to a screeching halt in college when I encountered severe gaps in my knowledge that made Calculus my hardest class. I had missed some very important lessons from earlier on that were huge hangups. It's hard to differentiate an equation if you aren't solid on how to factor the variables.

I had the same reaction as you. I think MOOCs are lacking in more than just this area. I'll use myself as an example:

I used MOOCs to learn various programming languages, simple database queries (insert, update, select with join, etc.) and how to use APIs with JSON. Lots of fun stuff. But when I came to university I started being taught all kinds of math and CS stuff through difficult assignments and tight deadlines that I never would have done on my own.

Yes, you technically can learn a lot of math and other things through MOOCs but it's not always as fun. The advantage of university is that they give you a set of required courses for a major and you have to take them, whether they're fun or not. Sometimes the courses which are the least fun turn out to be extremely valuable later on.

Data from an external source is usually fuzzy and requires some communication to reduce the fuzziness so that it can be properly interpreted. That involves communicating to understand why it's represented a specific way or negotiating to change the representation to better fit your needs.

The same is true for the output of your work.

That's a part of it, which doesn't come up in MOOCs at all (specifically, collaboration in a corporate environment within the software development lifecycle, and negotiating with project stakeholders about specs, requirements, and results).

I also didn't learn anything about that in school, but I did from my coding bootcamp, so if anything I think that is a general criticism of the education system.

I think the soft skills are essential to success, but can be easily learned by someone who is technically proficient.