The most effective way is likely ostracism, or some other penalty based on reputation. If someone doesn't pay their debts, people just stop trading with them. I'd posit that most people are interested in restitution, rather than punitive measures, which would be the primary goal of a DRO -- making the victim whole again. This is why DROs would likely function similar to an insurance agency.

The non-aggression principle is a central tenet in ancap philosophy which would preclude violence in most cases, however, it is permitted in self defense of ones person or property. Still, most people who subscribe for a DRO probably aren't going to want to pay the high costs for a standing militia. Would you voluntarily pay taxes for the War on Terror, or the numerous other military boondoggles across the globe, if you had a choice and knew the actual costs?

This ventures into the territory of private defense agencies, which decentralize security services, and is a deeper subject. If you want to dive in, I'd suggest Michael Huemer's book "The Problem of Political Authority: An Examination of the Right to Coerce and the Duty to Obey" in which he debunks the Hobbesian war of all against all scenario. "Practical Anarchy" by Stefan Molyneaux is also a decent (and free to read) overview of how ancap voluntarism could work, but Huemer's arguments are more comprehensive.