Door to door, I can beat the airlines on most trips under 1000 miles. I don't need to get to the airport 90 minutes ahead of departure, go through security, wait for boarding to complete, wait for pushback, connect through a hub, wait for bags, and can often land at an airport closer to where I want to be and have a rental car or cab waiting. The fact that cruise speed is only 215 mph/350 kph is made up for by all the slippage inherent in airline travel.
Even in places where you can't land on roads, there are WAY more airports than you probably think in the US. It's about 19K, with a little over 5K of those being public use. (Even the private ones are often accessible with just a coordinating phone call.)
Of those, fewer than 400 are "Primary" airports (defined as having scheduled commercial service and more than 10K enplanements [commercial passengers outbound] per year).
This is just too slow to get the main benefit of air travel, which is being able to go at barely subsonic speeds without handling being a massive headache. Of course you will have worse handling at barely subsonic speeds, due to any large deflection going transsonic and having the potential to result in loss of control. But that can easily be handled electronically, especially if the plane is at least somewhat aerodynamically stable, i.e. won't require constant corrections to not go into stall (like some modern fighter jets do).