Median is better than mean as a metric for "average goodness of driver". To take a median, you only need a strict ordering; to take a mean, you need to quantify. Any reasonable method of quantifying how good a driver someone is will be heavily influenced by environmental factors. The worst driver on an infinite, featureless plane would be involved in fewer accidents than the best driver in, say, New York.

Still, it could be even higher than 50% wrong, since sometimes the most capable people underrate their ability: http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

Yeah, if you take average=mean, then well over 50% of drivers are above average.