At least at the University level it seems like free speech was an important factor for deciding to allow hosting somebody with such an incendiary message. Here's the chancellor's statement: http://news.berkeley.edu/2017/01/26/chancellor-statement-on-...
For example, would you describe the Black Panthers liberals? Personally I wouldn't. They may share some objectives with liberals - equal treatment of a minority - but methods matter. I would no more call those rioters liberals than I'd call white supremacists 'conservatives'.
Unfortunately, most of the rest of the country thinks otherwise.
There is also the argument that fascism is an existential threat to liberal democracy. See the paradox of tolerance.
And yet they weren't throwing rocks at Bernie because his followers referred to black people as low-information voters. Nor were they protesting Hillary because her followers referred to Obama as a boy.
Do you really not see the double standard?
Do you seriously, honestly believe that that is only or even primary reason people might have for protesting Milo?
Yes. The reason most people are giving for protesting him is that he's racist, and the fact that one of his followers called Leslie Jones an ape is always cited as the reason for why he's racist.
If you don't believe me, look at the reasons the ADL gives for why he's racist:
http://www.adl.org/combating-hate/domestic-extremism-terrori...
There's a million reasons to protest this guy. You're missing the forest through the trees, by focusing on only one (That you consider absurd or hypocritical).
Bernie Sanders also didn't speak out in support of a muslim ban, or building a wall, or deporting millions of people... But Milo sure did.
If he held his rally on the lawn, that would be a different story.