What you have said is akin to saying that if the government banned the publishing of certain books that wouldn't be an impairment on the written word because people would still be free to write whatever they wanted down in their own personal notebooks.
> Reducing the ability of someone to broadcast their speech is an impairment.
Not necessarily; it all depends on perspective. Campaign finance restrictions would (relatively) enhance the ability of a not-so-wealthy person to broadcast their speech, because their message wouldn't be so drowned out by the broadcasts of the wealthy.
The amendment says that the government has to stay out of the way.
The First Amendment isn't the sum total of American law. I'm not a constitutional lawyer, but I'm pretty sure there are constitutional powers meant to provide effective democratic elections.
The tension here is between those two interests. I think it's worth considering the idea that a too-absolutist interpretation of the First Amendment (in regards to spending) may interfere with the nation's ability to provide effective democratic elections. Both are important priorities.
Also, it's not like the Constitution is perfect, timeless document. If it doesn't do enough to ensure effective democratic elections in light of its other provisions, then changes should be considered to fix that oversight.