Right. But without an activation function, the matrix multiplications would just compose into a single matrix for the whole net. I guess the floating-point rounding brought up above is supposed to serve as an activation function for this purpose, though I haven't looked into it.

But can't you typically do the activation as a separate step? Where you first incorporate all of the inputs, then decide activation?

I'm not sure quite what you mean, but you can do the linear part of one layer all together, then the nonlinear part, etc.:

    for weight,bias in layers:
        a = sigmoid(np.dot(weight, a) + bias)

So my question is how large is the linear part? Granted, I was hoping it was big enough to bring these algorithms into consideration. Looks like I'm right in that they are larger than 2x2. However, they are still not 8kx8k. :(

They can be pretty big! I don't know what's normal these days, though.