I would argue that it is the right's embrace of authoritarianism rooted in nativist and nationalist (some might even say racist) sentiments that pushed libertarians to the left, which is traditionally the sphere of people who have been historically oppressed or just want to be left alone.