But it's not money==speech. More like often times restricting money is restricting speech. That doesn't necessarily mean money is exactly speech.

Which is why the government still restricts campaign donations. It just can't restrict money when that money is being used for actual speech.