Re:“Less than one”-shot learning
by ampdepolymerase on Sunday October 18, 2020 @04:05PM
(#24820676)
It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.
"Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
Re:“Less than one”-shot learning
by heyitsguay on Sunday October 18, 2020 @09:49PM
(#24822707)
I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
Re:“Less than one”-shot learning
by ampdepolymerase on Sunday October 18, 2020 @11:57PM
(#24823213)
For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns.
The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.
So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?
mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.
Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?
Re:“Less than one”-shot learning
by freeone3000 on Sunday October 18, 2020 @04:24PM
(#24820844)
The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.
To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.
It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.
At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.
Ok, we've replaced the title with the first part of the subtitle.
Re:“Less than one”-shot learning
by gameswithgo on Sunday October 18, 2020 @04:04PM
(#24820670)
Is this process net faster, or requiring of less memory?
> In a previous paper, MIT researchers had introduced a technique to “distill” giant data sets into tiny ones, and as a proof of concept, they had compressed MNIST down to only 10 images.