News for hackers. Stuff that matters.
Re:Why is so much memory needed for deep neural networks? by jacquesm on Wednesday March 22, 2017 @08:30PM (#13936471)
If this is the sort of thing that keeps you awake at night: Nvidia has a $700 card (GTX1080ti) with 11G RAM and a GPU that is a slightly modified version of the GP102, which powers the more expensive Titan X.
Re:Why is so much memory needed for deep neural networks? by deepnotderp on Wednesday March 22, 2017 @08:38PM (#13936510)
Perhaps more efficient memory use is the answer instead. This article has mentioned most of the published techniques.

As for the 1080 Ti, a better option might be the P100, which can have up to 16GB.

Re:Why is so much memory needed for deep neural networks? by gwern on Wednesday March 22, 2017 @08:44PM (#13936536)
A better option... if someone else is paying for it. It is literally ~10x the price of a 1080ti ($700 vs a suggested $7000).
Re:Why is so much memory needed for deep neural networks? by BrailleHunting on Wednesday March 22, 2017 @09:14PM (#13936677)
The P100 has much better double-precision performance, but it's single-precision perf doesn't justify the cost in most instances.

The DGX1 (w 8 P100's) is another example of an overpriced, turnkey money extraction "solution."

Also worth noting it's hard to get a Ti non-founder edition, which has a hefty premium on it for the privilege of being first. The regular 1080 non-FE cards are probably the best bang/buck right now.

Folking doing ML/AI/DL, or any HPC really, gotta profile their apps to find bottlenecks and root out resource waste: single prec math? ram? algorithm complexity? storage iops? network latency? etc. because throwing money at a problem is no substitute for having a clue how to use money, time and electricity wisely.

Re:Why is so much memory needed for deep neural networks? by llukas on Thursday March 23, 2017 @12:27AM (#13937373)
Compared to what it can replace DGX-1 is not overpriced in any way.

Feel free to invest your time and money into building and debugging issues on your own system. Companies which pay big $$$ to their data scientists have different incentives than regular users - you do not want to waste time of your expensive staff on chasing hardware/software issues.

Re:Why is so much memory needed for deep neural networks? by Roritharr on Thursday March 23, 2017 @12:58AM (#13937487)
129000$ for the DGX-1, for anyone interested.

Saves you debugging your way to a workable 8*1080ti, when you look at it from this angle. Not sure if being able to swap out the cards is what makes building them yourself worth it in addition to the cost saving.

Re:Why is so much memory needed for deep neural networks? by modeless on Thursday March 23, 2017 @02:42AM (#13937807)
P100 has double rate FP16 which makes it twice as fast, so make that 16x 1080Ti. Also those won't fit into one machine, so now you're building a cluster, and what are you going to use for the interconnect? You won't get linear scaling, so make that more like 24x 1080Tis plus extra development effort for scaling on a cluster, if your problem even scales that way. Now account for the power usage...
Re:Why is so much memory needed for deep neural networks? by llukas on Thursday March 23, 2017 @03:58AM (#13938029)
What interconnect do you have between those 8*1080ti? If your problem doesn't require communication then it may make sense.

If you require non-trivial communication no PCIe setup would beat NVLink communication bandwidth on DGX-1.

Re:Why is so much memory needed for deep neural networks? by jacquesm on Thursday March 23, 2017 @07:15AM (#13938788)
> 129000$ for the DGX-1, for anyone interested.

If you are allowed to buy one. Nvidia doesn't have nearly enough chips to meet demand, the bulk of their production has been earmarked for quite a while ahead, what little is left over and makes it into the DGX-1 is sold to very few hand picked customers.

Re:Why is so much memory needed for deep neural networks? by ansible on Thursday March 23, 2017 @03:54AM (#13938017)
There is also the unified memory on the DGX-1, which is supposed to make streaming data to/from the GPUs a lot more efficient, as well as enable much larger data sets.
Re:Why is so much memory needed for deep neural networks? by cameldrv on Thursday March 23, 2017 @02:19AM (#13937731)
The P100 also has 2x the performance on FP16 and NVLink, which is about 10x faster than PCIe.
Re:Why is so much memory needed for deep neural networks? by jacquesm on Wednesday March 22, 2017 @08:41PM (#13936521)
> As for the 1080 Ti, a better option might be the P100, which can have up to 16GB.

$4K or so, vs $700?

Re:Why is so much memory needed for deep neural networks? by deepnotderp on Wednesday March 22, 2017 @08:48PM (#13936559)
Most of the major labs I've seen pay six figures easily to their top researchers. Good compute is one of the biggest reasons why researchers would want to work somewhere. Skimping on compute seems foolish at that stage.

Here's a half-joking, half-serious take on the matter: https://twitter.com/deliprao/status/842636496666419200

Re:Why is so much memory needed for deep neural networks? by seanmcdirmid on Wednesday March 22, 2017 @11:39PM (#13937246)
> Most of the major labs I've seen pay six figures easily to their top researchers.

Even a normal programmer easily makes six figures, I hope they are doing better than that given the demand in this space.

Re:Why is so much memory needed for deep neural networks? by deepnotderp on Thursday March 23, 2017 @12:09AM (#13937319)
That was a conservative estimate.
Re:Why is so much memory needed for deep neural networks? by posterboy on Thursday March 23, 2017 @01:42AM (#13937611)
I thought you might have meant monthly
Re:Why is so much memory needed for deep neural networks? by taneq on Wednesday March 22, 2017 @10:32PM (#13937004)
Yeah but is that P100 better than five or six 1080Ti cards?
Re:Why is so much memory needed for deep neural networks? by deepnotderp on Thursday March 23, 2017 @12:09AM (#13937321)
Because parallelization is so difficult, yes.
Re:Why is so much memory needed for deep neural networks? by milesrout on Thursday March 23, 2017 @04:31AM (#13938150)
If you are using a GPU, parallelisation is not difficult.
Re:Why is so much memory needed for deep neural networks? by jacquesm on Thursday March 23, 2017 @05:55AM (#13938463)
I believe you may be missing something. Even on a GPU there are synchronization points in some algorithms and data needs to be passed from one processor to another once you spill over the boundary of what can be held in the RAM.

Also, some parts of an algorithm may not be parallelizable at all.

GPU parallelization is only 'not difficult' if:

  - all your data fits on a single GPU

  - your code is embarassingly parallel
 
  - for the total duration of one computation
Bonus if you do can use the output of one computation as the input of the next. In all other kinds of computations the usual bottle-necks apply.

https://en.wikipedia.org/wiki/Amdahl%27s_law

Re:Why is so much memory needed for deep neural networks? by deepnotderp on Thursday March 23, 2017 @10:03AM (#13939783)
I was talking about multi-GPU parallelization. Multi GPU SGD parallelization is not easy. Single GPU parallelization is trivial of course.
Re:Why is so much memory needed for deep neural networks? by mijoharas on Thursday March 23, 2017 @08:22AM (#13939121)
My understanding from this article is that for a greater GPU ram size you will need to increase the minibatch size which has a negative impact on the models ability to generalize[0].

[0] https://arxiv.org/abs/1609.04836