News for hackers. Stuff that matters.
Re:Why is so much memory needed for deep neural networks? by milesrout on Thursday March 23, 2017 @04:31AM (#13938150)
If you are using a GPU, parallelisation is not difficult.
Re:Why is so much memory needed for deep neural networks? by jacquesm on Thursday March 23, 2017 @05:55AM (#13938463)
I believe you may be missing something. Even on a GPU there are synchronization points in some algorithms and data needs to be passed from one processor to another once you spill over the boundary of what can be held in the RAM.

Also, some parts of an algorithm may not be parallelizable at all.

GPU parallelization is only 'not difficult' if:

  - all your data fits on a single GPU

  - your code is embarassingly parallel
 
  - for the total duration of one computation
Bonus if you do can use the output of one computation as the input of the next. In all other kinds of computations the usual bottle-necks apply.

https://en.wikipedia.org/wiki/Amdahl%27s_law

Re:Why is so much memory needed for deep neural networks? by deepnotderp on Thursday March 23, 2017 @10:03AM (#13939783)
I was talking about multi-GPU parallelization. Multi GPU SGD parallelization is not easy. Single GPU parallelization is trivial of course.