That said, I don't think you are correct, at all, when you say that money is the only thing that matters. Votes matter. For example: The Nov 2012 Los Angeles elections. I voted in those elections. Mayor Garcetti won the election with only 5.4% of the city voting for him [0]. That means that I voted for ~20 Los Angelenos with my single ballot. Basically, I voted for half my homeroom in High School. You cannot buy votes nor stuff ballot boxes at the exchange rate I was able to affect with my single vote. And this was a presidential election, where turnout is typically higher. For the majority of local elections, I was voting for nearly my entire graduating class!
Another good example was Eric Cantor's primary election in 2014 [1]. Despite being the House Majority Leader, and despite outspending his opponent not just 2 to 1, not just 10 to 1, but 40 to 1, Cantor LOST. And it wasn't close, he lost 45% to 55%, a 10 point difference. Cantor resigned as Majority Leader on July 31st.
Money does not count.
Your. Vote. Counts.
[0]https://en.wikipedia.org/wiki/Los_Angeles_mayoral_election,_...
[1]https://en.wikipedia.org/wiki/United_States_House_of_Represe...
Even if you erased state boundaries (if you don't, the actual number goes up a bit), that would lead to a minimum of 8,143 members in the House.
You'd, IMO, actually do more to improve quality of representation and less to make the House unwieldy if you kept the current number of districts and made each district a 5-member constituency with STV or a similar candidate-centered proportional method of election.
But that's not what partisans do by and large outside a few key ones, because in the end for most partisans it's not about policy proposals or trying to steer the gov't, it's about identity.