Re:Beating Floating Point at Its Own Game: Posit Arithmetic ...
by stephencanon on Sunday July 02, 2017 @07:12AM
(#14681987)
If you're bound by memory throughput, you can't go beyond a 2x speedup (there's 1/2 as much data to move in an 8b format, whether it's in vectors or matrices doesn't matter). I still don't see any reasonable expectation for 4x.
Re:Beating Floating Point at Its Own Game: Posit Arithmetic ...
by dnautics on Tuesday July 04, 2017 @06:37AM
(#14694933)
if your matrix contents are static during your rate-limiting step (as they are for most DL applications) your FLOPs scale with O(n^2) relative to your memory throughput on your vector component.
[a b, c d] dot [e, f]
is four multiplies
[a b c, d e f, g h i] dot [j, k, l]
is nine multiplies.
A couple of points: 2-4x faster means 2x faster in dot product based simd and 4x faster in matrix simd, assuming that your bottleneck is memory throughput.
The sigmoid function realistically isn't a bottleneck in general, but you gotta admit it is pretty cool to have a ~zero clock cycle approximation. (I've tried to rein in John a bit on this one)