Network is, in fact, a major player in the latency, and by being globally distributed, configuring Route53 appropriately, and integrating CloudFront CDN, a given web-app gets a boost that I doubt a faster computer can beat.
The EC2 network also has mysterious packet filtering on it that prevented IPSec tunnels from working correctly.
I've managed to leverage CDN and globally distributed servers for a fraction of the cost of Amazon services just fine, and I have the added benefit of 100% full control of all aspects of it, including the network.
A good analogy is that EC2 is like an interpreted language versus compiled - you can get a lot done and it's easier to get started, but if you're really serious about performance, you need to program in C.
Those are cheap services which improve speed dramatically.