Re:Limits on Matrix Multiplication
by abecedarius on Saturday September 01, 2018 @08:58AM
(#17891624)
Right. But without an activation function, the matrix multiplications would just compose into a single matrix for the whole net. I guess the floating-point rounding brought up above is supposed to serve as an activation function for this purpose, though I haven't looked into it.
[ Parent
| on HN ]
But can't you typically do the activation as a separate step? Where you first incorporate all of the inputs, then decide activation?
Re:Limits on Matrix Multiplication
by abecedarius on Saturday September 01, 2018 @10:47AM
(#17892074)
I'm not sure quite what you mean, but you can do the linear part of one layer all together, then the nonlinear part, etc.:
for weight,bias in layers:
a = sigmoid(np.dot(weight, a) + bias)So my question is how large is the linear part? Granted, I was hoping it was big enough to bring these algorithms into consideration. Looks like I'm right in that they are larger than 2x2. However, they are still not 8kx8k. :(
Re:Limits on Matrix Multiplication
by abecedarius on Saturday September 01, 2018 @01:34PM
(#17892945)
They can be pretty big! I don't know what's normal these days, though.