News for hackers. Stuff that matters.

A comment on Artificial Intelligence Software Is Booming, But Why Now?. Parent comment

Re:Artificial Intelligence Software Is Booming, But Why Now? by hooloovoo_zoo on Sunday September 18, 2016 @11:54PM (#12528754)
Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems. I think #2 is very important; they speed up both training and evaluation by a factor of 100 compared to a modern CPU.

None of these developments really explain why AI has become so popular though. Neural nets don't actually perform very well for most problems (image and speech problems being the major exceptions) compared to tree-based methods. I think it has more to do with the development of easy-to-use machine learning libraries as well as distributed processing libraries and hardware.

[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by kordless on Monday September 19, 2016 @12:33AM (#12528934)
Consciousness of Things is an inevitable conclusion for humanity. The singularity is upon us with the advent of man merging into machine. We are driving that growth to go to the next level.

Also, containerization.

[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by glial on Monday September 19, 2016 @01:00AM (#12529057)
Most of what we call 'AI' today is just fancy regression methods. No need to be so grandiose.
[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by archgoon on Monday September 19, 2016 @02:09AM (#12529328)
I'm pretty sure that comment was completely facetious. Note that The Singularity got lumped together with Docker containers.
[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by kordless on Monday September 19, 2016 @08:58PM (#12536120)
Don't forget CoT vis-à-vis IoT.
[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by R_haterade on Monday September 19, 2016 @02:38AM (#12529424)
I lol'ed. Thank you for this.
[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by eli_gottlieb on Monday September 19, 2016 @08:15AM (#12530735)
>Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems.

They're essentially an informed prior on image and sound problems, which vastly reduces the size of the parameter space and thus makes both vanishing gradient and overfitting less of a problem.

[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by jhj on Monday September 19, 2016 @12:33AM (#12528936)
> they speed up both training and evaluation by a factor of 100

The factor is more like 5-10x for truly optimized CPU versus GPU in fp32 (e.g., nnpack on CPU versus cuDNN).

[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by hooloovoo_zoo on Monday September 19, 2016 @12:46AM (#12528997)
I'm basing that number on https://github.com/jcjohnson/cnn-benchmarks. I'm sure the exact number depends on the cpu/gpu/algorithm.
[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by jhj on Monday September 19, 2016 @12:54AM (#12529030)
im2col + sgemm on CPU as in Caffe for instance is really slow; you are heavily penalized for extra memory traffic and the sgemm tile sizes are probably not well tuned for the problem size at hand.

At the roofline of performance, the difference in both mem b/w and arithmetic throughput between CPU and GPU is only 5-10x (for fp32, Pascal fp16 is a different story of course), and proper implementations on the CPU will get you there.

https://github.com/Maratyszcza/NNPACK

[ Parent | on HN ]
Re:Artificial Intelligence Software Is Booming, But Why Now? by dharma1 on Monday September 19, 2016 @04:11AM (#12529751)
nnpack is excellent - but also very new. Noticed it just got NEON optimisations for Arm- do you know if it's possible to use it together with OpenCL on mobile for combined CPU/GPU inference?
[ Parent | on HN ]