* * *

Supervised models would also require a lot more human labour, and the goal of most machine learning projects is to achieve cost-savings by eliminating human labour.

Up front, yes, but long term, I wholly disagree. A model that performs at 95% or higher will assuredly eliminate human work, no matter how many interns you enlist to label the data.

People have tried (and continue to try) that human-in-the-loop data growth. Basically any applied AI company is doing something like that every day, if they're getting their own training data in the course of business. It helps but it won't turn your bag-of-words model into GPT3.

Companies like Google have even spent huge amounts of time and money on enormous labeled datasets -- JFT-300M or something like that for computer vision tasks, as you might guess, ~300M labeled images. It creates value, but it creates more value for larger models with higher capacity.

I "have tried (and continue to try) that human-in-the-loop data growth" to enormous success, bringing logistic regression models to greater than 99% accuracy. And you can chain vectorization strategies to create more input features than simply a bag-of-words, like morphology, shape, etc. We (the software company that I work for) don't need GPT-3, because it is a specialized model geared towards generating human-like text. Most NLP problems are just parsing text for actionable information, and oftentimes, supervised models can be chained to create something far more effective towards your needs than trying to shoehorn a massive general-purpose unsupervised model into a specialized problem.

It's almost like we have no clue what we are doing with NN and are just tweaking knobs and hoping it works out in the end.

And yet people still like to push this idea that we will magically and accidentally build a superintelligence on top of these systems. It's so frustrating how deep into their own koolaid the ML industry is. We don't even know how the brain learns, we don't understand intelligence, there's no valid reason to believe a NN "learns" the same way a human brain learns, and individual human neurons are infinitely more complex and "learning" than even a single layer of a NN.

As someone in the ML industry, who knows many people in the ML industry, we all know this. It's non-technical fundraisers that spread the hype, and non-technical laypeople that buy into it. Meanwhile, the folks building things and solving problems plug right along, aware of where limitations are and aren't.

> It's almost like we have no clue what we are doing with NN and are just tweaking knobs and hoping it works out in the end.

No, we understand very well how NNs work. Look at PartiallyTyped's comment in this thread. It's a great explanation of the basic concepts behind modern machine learning.

You're quite correct that modern neural networks have nothing to do with how the brain learns or with any kind of superintelligence. And people know this. But these technologies have valuable practical applications. They're good at what they were made to do.

We've got lots of great tricks for making audio ml run fast (we need to produce 16k samples per second on a mobile phone CPU, and sound great), but I think they haven't back propagated to the image or language communities.

Interesting, any material you can share?