Re:Only Train Once: A One-Shot Neural Network Training and P...
by medymed on Friday July 16, 2021 @05:41PM
(#27861556)
As a hobbyist, I’ve wondered if the need for umpteen epochs just leads many nets to memorize datasets, especially when the performance jumps a lot from one epoch to another without much change during batches. It’s kind of disconcerting for those of us who don’t have millions of source images to train with.
Re:Only Train Once: A One-Shot Neural Network Training and P...
by wxnx on Friday July 16, 2021 @06:25PM
(#27861911)
I think the evidence is pretty much in on that -- namely, yes, if your data is too small, a reasonably large neural net (a.k.a. basically any computer vision model from the last 3-4 years) is perfectly capable of memorizing the training images.
The relative success of attacks on nets to extract their training data support that this happens in practice too.
Generalization performance as it stands now always has to be evaluated empirically.
Re:Only Train Once: A One-Shot Neural Network Training and P...
by dheera on Friday July 16, 2021 @07:59PM
(#27862507)
The normal way to combat this is to have a training set and a validation set, use the validation images in forward pass to measure the performance of the network as it is being trained, but NOT use images from the validation set in updating the training weights.
Basically, evaluate the performance of the network on the validation set, but train it on the training set, and adjust your network structure and hyperparameters accordingly. Networks that "memorize images" will perform poorly.
Re:Only Train Once: A One-Shot Neural Network Training and P...
by enkid on Friday July 16, 2021 @08:47PM
(#27862740)
How does a large company with million if images make sure there aren't very similar copies in not the training set and validation set? Seems like you could be accidentally over training if there's a lot of copies of very similar images. (For example, there's going to be a lot more pictures of the Taco Bell Chihuahua compared to other Chihuahuas, and possibly some of those pictures have enough similarity to make over fitting possible.
Re:Only Train Once: A One-Shot Neural Network Training and P...
by dheera on Saturday July 17, 2021 @05:25AM
(#27864593)
First of all having millions of images is of itself a good way to combat overfitting.
But yes, you touch upon a very important point, that is the dataset must be sufficiently diverse.
To be fair, humans have this problem as well; if I showed you an octagonal red sign that had the words "GO" inscribed in the middle you may still mistake it for a stop sign at first.
Re:Only Train Once: A One-Shot Neural Network Training and P...
by anonymousDan on Friday July 16, 2021 @07:25PM
(#27862307)
I see they compare against various pruning techniques in related work, but do they compare accuracy and/or parameter reduction in their evaluation against lottery ticket approaches (whatever the state of the art is there)?
Re:Only Train Once: A One-Shot Neural Network Training and P...
by avidmoon on Friday July 16, 2021 @07:34PM
(#27862372)
Short answer, as I'm on mobile: this is structured pruning, which is hardly comparable to unstructured methods like IMP (algorithm for lottery tickets)
Re:Only Train Once: A One-Shot Neural Network Training and P...
by ndronen on Wednesday July 28, 2021 @01:20PM
(#27986347)
In unstructured pruning, individual weights of a weight tensor are pruned with no constraints on their position in the tensor. In structured pruning, there are constraints on which weights are pruned. Think of pruning an entire output channels of a convolutional layer vs pruning arbitrary weights of the same tensor. Unstructured pruning allows for a greater percentage of the weights to be pruned, but unstructured pruning doesn't speed up a network on hardware accelerators, which favor dense matrix multiplications.
The relative success of attacks on nets to extract their training data support that this happens in practice too.
Generalization performance as it stands now always has to be evaluated empirically.
Basically, evaluate the performance of the network on the validation set, but train it on the training set, and adjust your network structure and hyperparameters accordingly. Networks that "memorize images" will perform poorly.
But yes, you touch upon a very important point, that is the dataset must be sufficiently diverse.
To be fair, humans have this problem as well; if I showed you an octagonal red sign that had the words "GO" inscribed in the middle you may still mistake it for a stop sign at first.