I think most ex-libertarians eventually realized that (most of the time) a smaller government just means a power vacuum that gets filled by other organizations that have the means but not the peoples' interests.