News for hackers. Stuff that matters.
Re:Techniques for Training Large Neural Networks by liuliu on Thursday June 09, 2022 @05:24PM (#31687085)
1. I missed this paper https://www.microsoft.com/en-us/research/uploads/prod/2018/0... when doing my own feature map compression research: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de.... Thanks for point it out!

2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).

Re:Techniques for Training Large Neural Networks by MathYouF on Friday June 10, 2022 @02:29AM (#31690752)
This is really intersting.

> In particular, model parallelism is efficient when the amount of computation per neuron activity is high (because the neuron activity is the unit being communicated), while data parallelism is efficient when the amount of computation per weight is high (because the weight is the unit being communicated).

In this case, for fully connected layers, the amount of computation per neuron is high because it is (fully) connected to and from every other neuron in the pervious and following layer, and therefore we want more GPU's to work on those in parallel, so it isn't a time bottleneck?

What does it mean then for to have a high "computation per weight".

>• Convolutional layers cumulatively contain about 90-95% of the computation, about 5% of the parameters, and have large representations. • Fully-connected layers contain about 5-10% of the computation, about 95% of the parameters, and have small representations.

Wouldn't fc's require more computation since each parameter has more incoming and outgoing values?

I don't doubt this is all accurate and I'm just missing some obvious intuition, but for the sake of learning feel free to explain it if you wish!

Re:Techniques for Training Large Neural Networks by radarsat1 on Friday June 10, 2022 @07:38AM (#31692663)
Convolutional kernels are matrix-multiplied over and over again in a sliding window across the feature map. The fully connected layer is matrix-multiplied exactly once. So the number of times each element of the weight matrix is used depends on the size of the layer for FC, but depends on the size of the feature map for Conv layers. Usually the feature map is much larger than the kernel, for the majority of layers, so the kernel weights are re-used a lot.

The other possible interpretation is that typically an FC is only applied on a reduced dimensional embedding, after reducing the input data dimensionality iteratively by several convolutional layers. So there is overall more computation done by Conv layers, to get that final embedding, which his then fed to one or two FC layers for taking global context into account -- so usually you'll find more Conv layers than FC layers in a given ANN architecture, thus they make up a larger percentage of the overall computation.

Re:Techniques for Training Large Neural Networks by liuliu on Friday June 10, 2022 @12:46PM (#31696527)
The paper is very dated. You have to put on the lens that at that time, the networks we are talking about are AlexNet (the author) and VGGNet. For these, FCN layers has much more parameters than CNN layers, and all-reduce for gradients on these FCN layers are not economical (BTW, that's why Transformers do mixed model / data parallelism as well, or do large batches, they have a lot parameters and want to do these all-reduce as late as possible (or avoid at all, with model parallelism)).

Otherwise the peer comment answered your question really well!

Re:Techniques for Training Large Neural Networks by rllin on Friday June 10, 2022 @03:18AM (#31691099)
"One weird trick" is still pretty much the first way to go for most recommendation systems that are large embeddings focused. see torchrec, nvidia's hugectr
Re:Techniques for Training Large Neural Networks by liuliu on Friday June 10, 2022 @12:47PM (#31696530)
Great! Will check out more. Admittedly, recommendation system is one that I have less knowledge about.
Re:Techniques for Training Large Neural Networks by mirker on Thursday June 09, 2022 @06:14PM (#31687633)
Seems similar to the Huggingface tutorial: https://huggingface.co/docs/transformers/parallelism.

What’s different?

Re:Techniques for Training Large Neural Networks by p1esk on Thursday June 09, 2022 @07:11PM (#31688138)
Why should it be different?
Re:Techniques for Training Large Neural Networks by mirker on Thursday June 09, 2022 @10:45PM (#31689395)
Why link to this one vs. huggingface?
Re:Techniques for Training Large Neural Networks by cosentiyes on Thursday June 09, 2022 @11:25PM (#31689628)
Why link to huggingface vs. this one?
Re:Techniques for Training Large Neural Networks by Der_Einzige on Thursday June 09, 2022 @01:44PM (#31684373)
What if you abandoned gradients/back-prop entirely? They knew that for reinforcement learning this was good clear back in 2017!

https://openai.com/blog/evolution-strategies/

Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!

Re:Techniques for Training Large Neural Networks by pchiusano on Thursday June 09, 2022 @01:59PM (#31684575)
You might like these posts on gradient free optimization:

http://pchiusano.github.io/2019-07-06/learning-without-a-gra...

http://pchiusano.github.io/2020-10-12/learning-without-a-gra...

I think you’re right that the area is underexplored.

Re:Techniques for Training Large Neural Networks by eismcc on Thursday June 09, 2022 @03:19PM (#31685591)
Has anyone tried using another NN as the “gradient” function? With the hope of making it less “blind?”
Re:Techniques for Training Large Neural Networks by rolisz on Thursday June 09, 2022 @04:13PM (#31686184)
DeepMind has a thing called synthetic gradients: https://www.deepmind.com/blog/decoupled-neural-interfaces-us....
Re:Techniques for Training Large Neural Networks by erwincoumans on Thursday June 09, 2022 @04:25PM (#31686331)
Black box optimization such as CMA-ES or Augmented Random Search (ars) are used by researchers, for example:

CMA-ES: https://blog.otoro.net/2017/10/29/visual-evolution-strategie...

ARS: https://arxiv.org/abs/1803.07055

Those optimizers are efficient up to certain neural network sizes.

Re:Techniques for Training Large Neural Networks by hungrigekatze on Saturday June 11, 2022 @12:27PM (#31705716)
I was curious as to what the 'community AI' research org's stances on distributed training of deep neural nets were so some weeks ago I stumbled upon Eleuther AI's FAQ page which was talking about how it was not a task that they were looking at due to various technological challenges:

Source: https://www.eleuther.ai/faq/

What about volunteer-driven distributed computing, like BOINC, Folding@Home, or hivemind? -Backpropagation is dense and sensitive to precision, therefore requiring high-bandwidth communication. Consumer-grade internet connections are wholly insufficient. -Mixture-of-experts-based models tend to significantly underperform monolithic (regular) models for the same number of parameters. -Having enough contributors to outweigh the high overhead is infeasible. -Verifiability and resistance to outside attack are not currently possible without significant additional overhead. In short, doing volunteer-driven distributed compute well for this use case is an unsolved problem.

---

Am really excited to see inroads being made in this field of active research and hope that all AI orgs - OpenAI, Eleuther, etc. - can take part in this in domain of much-needed (IMO) research.

Re:Techniques for Training Large Neural Networks by superb-owl on Thursday June 09, 2022 @01:41PM (#31684333)
I'm really excited to see folks starting to talk about parallelizing machine learning. The conversation has been dominated by GPU-friendly techniques - a classic example of "everything looks like a nail when you have a hammer".

I hope we start seeing more massively parallel training strategies (most likely with GPUs under the hood still)

Re:Techniques for Training Large Neural Networks by kristjansson on Thursday June 09, 2022 @01:59PM (#31684586)
> starting

This is a strange comment to see. The techniques in TFA have been at the heart of a the last few years of progress in large-scale language models, image generation, etc?

Re:Techniques for Training Large Neural Networks by rg111 on Friday June 10, 2022 @05:14AM (#31691772)
Dude, parallelization is exactly how non-trivial neural networks are trained since the beginning.

Parallel processing as in SIMD architecture is what makes Deep Learning possible now.

Parallel computing as in stacking graphics card is how literally everything is done everywhere- from big tech companies to not-so-well-funded data science departments in unknown unis.

You are so so off.

Re:Techniques for Training Large Neural Networks by ShamelessC on Thursday June 09, 2022 @04:36PM (#31686506)
Strange comment. May need to edit something? I can't parse it - parallel (e.g. CUDA) techniques have been popular since AlexNet.
Re:Techniques for Training Large Neural Networks by Tostino on Thursday June 09, 2022 @04:49PM (#31686666)
I parsed that as meaning something more like BOINC to distribute the work and piece together the final result.
Re:Techniques for Training Large Neural Networks by mr_toad on Thursday June 09, 2022 @07:30PM (#31688280)
Like TensorFlow or PyTorch?
Re:Techniques for Training Large Neural Networks by sydthrowaway on Thursday June 09, 2022 @03:21PM (#31685611)
Amazing