I think you underestimate how difficult it is to drive versus walking up stairs or walking a mile.

Just as an example: blind people can't drive at all, but they can walk miles pretty easily and safely. And visual impairments of various sorts are, IIRC, the number one disability among the elderly.

Slower reaction times are similar: they destroy the ability to drive, but they're not a huge deal for walking up a flight of stairs.