Re:Why is so much memory needed for deep neural networks?
by seanmcdirmid on Wednesday March 22, 2017 @11:39PM
(#13937246)
> Most of the major labs I've seen pay six figures easily to their top researchers.
Even a normal programmer easily makes six figures, I hope they are doing better than that given the demand in this space.
Re:Why is so much memory needed for deep neural networks?
by deepnotderp on Thursday March 23, 2017 @12:09AM
(#13937319)
That was a conservative estimate.
Re:Why is so much memory needed for deep neural networks?
by posterboy on Thursday March 23, 2017 @01:42AM
(#13937611)
I thought you might have meant monthly
Re:Why is so much memory needed for deep neural networks?
by taneq on Wednesday March 22, 2017 @10:32PM
(#13937004)
Yeah but is that P100 better than five or six 1080Ti cards?
Re:Why is so much memory needed for deep neural networks?
by deepnotderp on Thursday March 23, 2017 @12:09AM
(#13937321)
Because parallelization is so difficult, yes.
Re:Why is so much memory needed for deep neural networks?
by milesrout on Thursday March 23, 2017 @04:31AM
(#13938150)
If you are using a GPU, parallelisation is not difficult.
Re:Why is so much memory needed for deep neural networks?
by jacquesm on Thursday March 23, 2017 @05:55AM
(#13938463)
I believe you may be missing something. Even on a GPU there are synchronization points in some algorithms and data needs to be passed from one processor to another once you spill over the boundary of what can be held in the RAM.
Also, some parts of an algorithm may not be parallelizable at all.
GPU parallelization is only 'not difficult' if:
- all your data fits on a single GPU
- your code is embarassingly parallel
- for the total duration of one computation
Bonus if you do can use the output of one computation as the input of the next. In all other kinds of computations the usual bottle-necks apply.
Re:Why is so much memory needed for deep neural networks?
by deepnotderp on Thursday March 23, 2017 @10:03AM
(#13939783)
I was talking about multi-GPU parallelization. Multi GPU SGD parallelization is not easy. Single GPU parallelization is trivial of course.
Here's a half-joking, half-serious take on the matter: https://twitter.com/deliprao/status/842636496666419200