> Do you seriously, honestly believe that that is only or even primary reason people might have for protesting Milo?

Yes. The reason most people are giving for protesting him is that he's racist, and the fact that one of his followers called Leslie Jones an ape is always cited as the reason for why he's racist.

If you don't believe me, look at the reasons the ADL gives for why he's racist:

http://www.adl.org/combating-hate/domestic-extremism-terrori...

How about when he publicly doxxed and insulted a transgender student at UW-Milwaukee as another reason to protest him among many many other valid reasons? http://nymag.com/thecut/2016/12/milo-yiannopoulos-harassed-a...