I touched upon this a bit; I think that the police don't have as big of a deterrent effect as you'd think. Social structures such as "I don't want to get robbed, so I don't rob others" play a much larger role.
This is a large and complex question, and so I'm going to bow out to the FAQ, which has a pretty in-depth discussion of this topic: http://anarchism.pageabode.com/afaq/secI5.html#seci58
TL;DR, most crime wouldn't exist under anarchism, because many crimes simply aren't crimes anymore. There's no incentive for someone to steal my wallet, because a small piece of leather isn't really useful.
> Hasn't society been down this road, in both micro and macro scenarios, many times before?
We have, and we still haven't found a good answer. Are you saying petty theft is nonexistent today? Plenty of people still steal wallets.
For example, someone decides to kick your teeth in and steal your wallet. Without police, what's your recourse? Vigilantism? What prevents a rising escalation of retaliatory vigilantism? Hasn't society been down this road, in both micro and macro scenarios, many times before?