Right, but why would this be the least of my concerns? What's the greater concern here?

Getting robbed, kidnapped, raped, killed, etc. - I personally had a "ghost" cabbie try to swipe my credit card data using a card swipper, demand to take me to an ATM, etc. - lucky, nothing "bad" happened; after the fact, called in his ID number and it wasn't real. This is a bigger issues in some areas, Uber first introduced a panic button in countries were cabbie attacks were a larger issue, but believe they're even adding it to some cities in the US now.

Depends what you mean by "greater concern." Obviously getting robbed, kidnapped, raped, or killed are among the worst possible outcomes of any scenario, but when I think of "greater concern" I tend to consider the likelihood of each outcome and not just the desirability.