News for hackers. Stuff that matters.
Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by gambler on Monday February 25, 2019 @03:23PM (#19249069)
>A single GPU in 10 years might be able to smoke these 512 GPUs.

Firstly, it will not, because there are physical limitations to silicon-based computing and I don't think we will get a different kind in the next 10-20 years. Secondly, unlike commodity programming, AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem. For starters, any algorithmic improvement makes all further research and experimentation easier for everybody as opposed to a handful of agents who have access to ridiculously powerful hardware. I can list many other reasons, like global energy consumption and the need for learning in embedded devices.

I find it highly annoying that deep learning enthusiasts immediately turn into uber-skeptics whenever the conversation touches on other ML approaches. Makes me wonder how difficult it is to get funding for fundamentally new AI research in this climate.

Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by MasterScrat on Monday February 25, 2019 @04:04PM (#19249453)
> AI is an area where improving algorithm/learning efficiency is infinitely more valuable than figuring out how to throw more hardware at the problem

Improving algorithm/learning efficiency gets much easier when you can iterate faster.

Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by WrtCdEvrydy on Monday February 25, 2019 @03:33PM (#19249180)
For reference, 10 years ago, graphics cards were on a 65nm platform while today, we use 14nm graphics cards.

The physical limitations may be reached, but we'll just throw more cores unto the PCB to compensate.

Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by JustFinishedBSG on Monday February 25, 2019 @04:12PM (#19249495)
We can't even do that, the biggest GPUs today are at the reticle limits of foundries> They literally can't be made bigger. Of course we can still go the chiplet way but...
Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by frankchn on Monday February 25, 2019 @04:30PM (#19249639)
... but power requirements would put a limit to that as well. The RTX 2080 Ti already has a 250W TDP. Putting a couple of those on a single card and you are looking at >1000 W. Cooling and power becomes very hard as we are effectively trying to run and cool a space heater at the same time.
Re:GradientFlow: Training ImageNet in 1.5 Minutes on 512 GPUs by CyberDildonics on Monday February 25, 2019 @04:28PM (#19249617)
But what?