If it was straight ad delivery I'd agree that being closer to 10ms is achievable. But in the process of pulling in ad data, user profile data, running it through a selection algorithm that can be pretty complex for even a simple adserver, calculating and plugging in macros, gzipping, and doing that thousands of times per second per server, the latency tends to creep upward.

More/faster hardware can obviously help, but since everything's async from the browser's perspective you tend to run into higher costs and diminishing returns by pushing for the lowest latency possible.