Geographical representation means that Congress critters have direct responsibility to a well defined set of constituents. So you can't make up for closing a factory on one side of the state by making new friends on the other side of the state.

Furthermore, you have to decide which reps stay on when an election reduces the number of reps. Who makes that choice? In parliamentary systems, the party does. But in the US, elections are (mostly for the worse, imho) ultimately about the individual representative: voters have a chance to say 'THIS guy is an asshole' and vote him out, instead of giving the party discretion as to whether to drop the entrenched asshole or kick out the newcomer...