I'm recalling some discussions with scientists in remote robotic surgery a few years back where I was told that for some truly high precision operations, the latencies become unmanageable past a point. I'd be curious to know if the state of the art has shifted at all, or if we really are starting to hit the limits of systems that require a certain bar of reaction time and precision.

Following from that, I wonder what bar Driving has, and as such, what distribution of "Controllers" we'd need to be safely within that, and if that would allow it to be viable.

Client side prediction would help a lot. Online racing games seem to easily handle 300ms latencies.

I fully realize that the difference between virtual racing games and real live humans in hunks of metal is massive. I'm just saying that work on dealing with latency in situations which are more predictable and have larger margins for error than high precision operations have been worked on ever since QuakeWorld.

Indeed, I would say that the work done getting e.g. OnLive to work would be extremely relevant to people driving remotely: if you can be competitive in an arbitrary FPS game streamed to you across the continent—in a direct adversarial contest of reaction time against people playing locally on their own hardware—then I'm pretty certain you can drive a car using streamed data. Especially since, unlike with games, drivers won't demand "excellent 1080p graphics and HD sound"; just something usable to get around. The car-side computer can do a lot of pre-processing of its own outputs to get the streamed data rate down to something useful.

The real problem, I think, would be making sure the (presumably cellular) connection to the car doesn't drop or suffer from latency spikes. If the car companies teamed up with the cell companies, there'd probably be big pushes to put 4G or WiMax microcells on every power-pole running down streets and highways, and even more in cities, to ensure cars remain online.