Frankly I think we'd benefit more, faster even from just better standard libraries of really simple methods in ways that makes it more obvious how to take advantage of them.

E.g. there are so many scenarios where even really basic statistical approaches like bayesian models can provide drastic improvements over what people tend to do, but most developers I've worked with don't know how to take advantage of even that and/or don't know when they can use them or how to communicate to stakeholders what capabilities are available.

There will be ridiculously many low hanging fruits in that area for years to come.

I'm trying to start simple on my journey towards thinking more like a statistician and have long suspected this would be the case at many companies. Besides Bayesian models, do any other common methods seem to be ignored as often? Any suggestions resources that a newbie on the scene might use to educate themselves?

One simple method that is often good enough is k-nearest neighbors. Basically computing the k most similar training values to a test value and then averaging their outputs to get a prediction.

Thank you! I'll be sure to dig into k-nearest more as well.

I would be interested in hearing about some examples. I'm one of those who are not yet taking advantage of these potential drastic improvements you are referring to.

A typical example is improving ranking in (custom/site specific) search. E.g. a client of a company I used to work at spent a huge amount with them having developers manually tweak search results. To add to it, they had multiple different stakeholders (e.g. sales team, executive team, their customers) who had different ideas about what should rank highly for different queries.

But they had a very simple metric right in front of their nose:

Their "product" was bookings at their customers restaurants. They were paid per cover.

One trivial improvement over the constant manual tweaks would be to simply record the probability of a user choosing to book at a given restaurant if that restaurant is present in the search result for a given term or set of terms.

Once you have that data, you can use Bayes theorem to take a set of tokens for a search and produce a list of probabilities that a given restaurant will be a good match, and rank accordingly. And you don't even need to understand Bayes theorem (though as someone who usually don't like maths papers, Bayes paper is remarkably understandable with very little mathematical background) - you just need to be aware of it and be able to use a simple library.

[you'll want to do some tweaks to account for the difference that position in the result makes]

There'll be an endless stream of improvements and more advanced methods you can use, but the beauty of applying Bayes theorem to data like this is that it is very simple, and you can explain to people why it ends up ranking the way it does very easily (the action you record, such as the decision to make a booking, serves as an "upvote" effectively).

There are lots of more advanced approaches you can take if you have the time and the skill and sufficient data, but the above works reasonably well with very little effort and is very often a substantial improvement over whatever manual attempts people make at guessing how the ranking ought to be. And it can be applied not just to search results, but to pretty much anything where you can derive an input set of terms (be it e.g. categories, or a paragraph from the text on the page) that has some likelihood of having some relation to how people will like another item, and where you can record an action against a related item as a "vote" for that connection.

Of course you can train deep learning models on small datasets. Where did you hear otherwise?

Try yourself at https://nanonets.com/

From the site, "large pretrained models + your data."

...it's not really a small dataset then, though.

The question is, is the power of deep nets constrained to tasks where you have a huge amount of data.

there are a large amount of applications where you can use big models trained on generic tasks where the data is free and available to everyone, and then fine tune to your specific more limited data.

let's say I need a person detector for my cctv camera. I use a pretrained ResNet ImageNet model distributed by Google, that was trained on a super large dataset identifying things like animals, furniture and whatnot on a huge free dataset, and then train it just a bit more on my specific task.

this allows to create very performant networks with very little task specific data.

He didn't say you couldn't, he said that's what Google means.

Cheers for the assist, but I really did mean that it's impossible to train deep learning with small datasets (also, I'm a "she" :).

yorwba below explains why transfer learning is not a way to train a deep net with small data sets. Rather, it's a way to re-train an already trained deep net. If you don't have a deep net trained on your target domain, then what?

Software devs are used to having access to libraries that make our lives simple. You don't have to invent a new language and write a compiler for it everytime you want to code, because java etc. That is the hope of services like the one discussed above.

However, because trained deep nets are very task-specific and it's virtually impossible for a small developer team with few resources to train one from scratch to reach state-of-the-art performance, developers who choose to use pre-trained deep nets as a library are perpetually dependent on Google, or whatever big company decides to offer its deep nets as a service, to determine the range of uses of their re-trained nets.

My understanding is that Google very much intends Cloud AutoML Vision usable for small datasets. To me "deep learning ... inherently impossible to train small" sounds entirely confused.

The whole advantage of deep learning comes from the large number of parameters that can fit very complex feature distributions. If you don't have a lot of data and try to train a deep net on it, you're just going to overfit.

Of course there's the opportunity for transfer learning where you try to adjust a pretrained model slightly for a different task, but that only works if the model already mostly solves your problem. (e.g. it picks up on the necessary features, but the class you want is not one of the possible outputs.)

If your data is unlike any popular task with large datasets available, (say because you're trying to predict something from a small in-house database), transfer learning can't work (because there's no knowledge you could transfer) and you are better off with simpler models.

>> make it possible to train deep learning on small datasets

Looks like Baidu's new voice cloning tech is able to do tricks like make a British man sound like an American woman with just a few seconds worth of audio to train on. Apparently the previous version needed at least half an hour of audio to train.

https://thenextweb.com/artificial-intelligence/2018/02/26/ba...

> That capability means either abandoning deep learning as inherently impossible to train small, or spending major resources to make it possible to train deep learning on small datasets with little compute.

> Is Google working on this problem? Is anyone?

Yes, but this is very much out of fashion at the moment (just as NNs were out of fashion for about three decades). Look at the work of Gary Marcus at NYU or Michael Whitbrock at IBM.

(We are also, and think we're farther along than anyone else but aren't talking about it publicly yet so no need to believe me. There are a few others in a similar state)

For smaller data sets I recommend checking out Gamalon (no affiliation, just think its cool) https://gamalon.com/

Not to sound like a judgmental jerk, but how is this cool? It's just a very vague landing page.

It's cool because it can train on very small data sets and recognize multiple images in one frame. For example, draw a chair, then draw a lamp next to it. It will recognize both the chair and the lamp not having been fed millions of images of chairs and lamps together.

Check out the demos by Gamalon: https://gamalon.com/technology/

(I have no affiliation to Gamalon)

It's called fine tuning.