And yet they weren't throwing rocks at Bernie because his followers referred to black people as low-information voters. Nor were they protesting Hillary because her followers referred to Obama as a boy.
Do you really not see the double standard?
Do you seriously, honestly believe that that is only or even primary reason people might have for protesting Milo?
Yes. The reason most people are giving for protesting him is that he's racist, and the fact that one of his followers called Leslie Jones an ape is always cited as the reason for why he's racist.
If you don't believe me, look at the reasons the ADL gives for why he's racist:
http://www.adl.org/combating-hate/domestic-extremism-terrori...
There's a million reasons to protest this guy. You're missing the forest through the trees, by focusing on only one (That you consider absurd or hypocritical).
Bernie Sanders also didn't speak out in support of a muslim ban, or building a wall, or deporting millions of people... But Milo sure did.
If he held his rally on the lawn, that would be a different story.
There is also the argument that fascism is an existential threat to liberal democracy. See the paradox of tolerance.