News for hackers. Stuff that matters.

A comment on How to use Deep Learning when you have Limited Data.

Re:How to use Deep Learning when you have Limited Data by sarthakjain on Thursday February 02, 2017 @05:10PM (#13555288)
Machine Learning and AI seem to be in vogue but become tough to implement unless you have boatloads of data. We've personally had multiple frustrating experiences over the last ~7 years of trying to solve problems using ML. In almost all the cases we failed to ship due to lack of data. Transfer Learning is a major breakthrough in ML where companies with little data can also build state of the art models. Unfortunately not enough people know about it. We are trying to do our part to make it easier to use Transfer Learning as well as increase awareness about it.

Using Transfer Learning we can build a model to identify cats and dogs in images with a few (<100) images as compared to the few thousands it would take before.

To make Transfer Learning easy we are building https://nanonets.ai that has multiple pretrained models that can be augmented with your data to create state of the art models. We are currently in the process of building our first few models. Image Labeling and Object Detection (in Images) work with a few Text based models coming up in the next few weeks.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by chriskanan on Thursday February 02, 2017 @05:56PM (#13555765)
Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Thursday February 02, 2017 @06:00PM (#13555792)
Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by qxcv on Friday February 03, 2017 @12:31AM (#13557927)
Google Trends can tell you a lot of things :)

https://www.google.com/trends/explore?date=2014-01-01%202017...

More seriously though: others have pointed out that finetuning is pretty popular in some subfields, but it's just one hammer in a of a whole toolbox of techniques which are necessary to make neural nets train (even when you have a tonne of data). Standardisation, choice of initialisation, and choice of learning rate schedule all come to mind as other factors which seem simple, but which can have a huge impact in practice.

Of course, each tool has its limitations. The most obvious limitation of finetuning is that you need a network that's already been trained on vaguely similar data. Pretraining on ImageNet is probably not going to help you solve problems where the size of objects matters, for example, because most ImageNet performance tends to benefit from scale invariance.

I wish you luck with nanonets.ai, but I think it's irresponsible to market this as the "1 weird trick" to bring data efficiency to neural nets.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by averagewall on Thursday February 02, 2017 @09:22PM (#13557105)
That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained network?

Personally, I'm a hobbyist and I don't want to know about these shortcuts until I start to need them - which is a stage I might never reach. People who've progressed far enough to need them are probably far fewer than those who are just curious what these words mean.

Another possibility is the words "transfer learning" might be more generally meaningful outside the ML field than the other search terms on the graph, so most of the searches for it are really schoolteachers or something else.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Thursday February 02, 2017 @09:40PM (#13557196)
Spot on. If you take that argument a step further that means an average developer who is not a data scientist or ml researcher might not know about it. Which implies a super easy to use dead simple technique which is used by most researchers is not available to the common developer even though it is easy enough for them to use.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by kyloon on Thursday February 02, 2017 @09:23PM (#13557110)
It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by almondai on Friday February 03, 2017 @03:40AM (#13558587)
Transfer learning in ML refers to the general idea of taking a model trained for one domain and applying it to another. However this article seems to only focus on feature mapping, ie. breaking down images into features using hidden layers of ImageNet models. In this case, the pretrained model is only acting as a feature extractor because it is not trained to maximize the embedded distance between the classes you are trying to differentiate.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Friday February 03, 2017 @04:56AM (#13558806)
Your right the article doesn't do justice to all of the transfer learning techniques. I cut out large portions of the original draft for brevity. Will write a longer post referencing all possible techniques, hopefully it won't turn into a review paper.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Thursday February 02, 2017 @09:42PM (#13557220)
Your right about excitement been generated from new techniques in the research community. Also if you look at Prisma, Google photos or any number of applications that have gained mass popularity. They are just implementations of existing networks and techniques. The paper on style transfer made Prisma possible, commercialization usually has been a few steps behind research.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by prats226 on Thursday February 02, 2017 @09:31PM (#13557152)
True that most excitement in the field has been in finding new architectures and building general purpose AI. However I would like to point out that people often underestimate potential of transfer learning. We have started solving real problems companies are facing right now because of lack of enough data to be able to train an accurate model themselves.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by Ayyar on Friday February 03, 2017 @01:06AM (#13558068)
This is Deep Learning 101 level material, rather than an advanced insider technique.

It's even listed in the Tensorflow tutorials: https://codelabs.developers.google.com/codelabs/tensorflow-f...

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Friday February 03, 2017 @02:39AM (#13558397)
While I agree that you could build a demo in Deep Learning 101 that could work for some small set of examples, I disagree that this is 101 level material.

Facebook just released: https://techcrunch.com/2017/02/02/facebooks-ai-unlocks-the-a...

You could also call this Deep Learning 101. But it really isn't because building a usable platform that works at scale actually delivers performance and solves problems is a lot tougher than what can be taught in an intro to Deep Learning 101 course.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by ChaitanyaSai on Friday February 03, 2017 @04:47AM (#13558780)
Just poked around a bit with your API, and the learning with just 25 samples is impressive! And the getting training samples from the web is a great touch. But that 25 sample number seems too low for classes that are "closer" together? How do you quantify if you have done a good job on training or if you need some varied samples?
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Friday February 03, 2017 @04:59AM (#13558822)
Thanks for trying us out! Internally we have validation metrics which have a number that says how good a model is at class separation. One naive way to do this entropy (-plogp). We are planning on exposing this to users soon. So once you create a model you'll receive feedback as to how "good" we think it is. In case it's not working well we might ask you for more data (we hope we don't need to do this too frequently)
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by prats226 on Friday February 03, 2017 @01:19AM (#13558122)
Transfer learning is fairly well studied. However we have seen lot of companies facing problems that can be solved using this technique but can't because of lack of knowledge, pretrained models availability, engineering challenges involved. We are just trying to make the process easier for them.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by ovi256 on Friday February 03, 2017 @08:37AM (#13559671)
Another great example is the last post on the keras blog [1] "Using pre-trained word embeddings in a Keras model". You take advantage of a large pre-trained network for a text classification task.

The OpenFace face recognition library also offers this technique. You take advantage of their large pre-trained network for face embedding: transforming a face into features distinct enough for classification. You then train another few layers for recognizing your own samples.

1: https://blog.keras.io/using-pre-trained-word-embeddings-in-a...

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by scottyli on Thursday February 02, 2017 @08:59PM (#13556999)
What are the downsides of transfer learning? How can it fail?

And do you just arbitrarily select the "cut off output layer" for the pretrained model when retraining with your own data on new layers?

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by prats226 on Thursday February 02, 2017 @09:15PM (#13557070)
One way it can fail is that your model might overfit your data if number of parameters you are training are way more than data you are using for training and if regularization techniques are not used. We don't arbitrarily cut-off layers from pretrained model. We check at what layer, output features are not specific to problem for which pretrained model was trained. At this layer, you can stitch a nanonet and build a model for your data
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by nl on Friday February 03, 2017 @12:28AM (#13557919)
Transfer learning works pretty well for image classification related tasks.

Some other areas are much more challenging. For example, in natural language processing tasks you will sometimes see some benefit from using pretrained embeddings, but it is very task and model specific. There's some exciting work going on in this area though.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by prats226 on Friday February 03, 2017 @12:42AM (#13557979)
Yeah correct. Even with text, there is some exciting work going on. We are in process of building a text based model and should put it up in this week that should work for multiple problems.
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sfifs on Friday February 03, 2017 @12:04PM (#13561246)
what do your false positives and false negatives rates look like with this approach compared to other classifiers?
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by deepnotderp on Friday February 03, 2017 @02:01AM (#13558272)
I'm inclined to agree with the other commentators here, this is pretty much learnt in "Deep Learning 101" (cs231n)
[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by sarthakjain on Friday February 03, 2017 @02:15AM (#13558307)
Great feedback, I really appreciate you taking time to read the post. Though I would like to make a few notes regarding what we are trying to achieve and some improvements we've made:

1) Our target audience is someone who hasn't taken Deep Learning 101 but wants to solve a problem

2) We are focusing on users who don't want to setup their own deep learning machines and don't want to learn how to use tensorflow/kerras/caffe/theano and spend time maintaining their own boxes, they don't want to spend engineering effort in ensuring slas and uptime along with scalability

3) We have made improvements in both the model we use for our product and the way it is retrained. It's not the same as the tensorflow example

4)The model has a different dataset than ImageNet and provides additional value in being better suited to certain tasks.

Worst case we have something nobody wants and that's valuable insight in itself. In the best case we have made something that people learn in Deep Learning 101 that can now be used by anybody without spending time and get straight to solving problems.

[ Parent | on HN ]
Re:How to use Deep Learning when you have Limited Data by deepnotderp on Friday February 03, 2017 @02:57PM (#13562738)
Fair enough, that makes sense, but I kind of have an allergic reaction to AI hype after being burned a few times :)

OTOH, your webpage makes it pretty clear what you actually do, so props to you for that!

[ Parent | on HN ]