Why not allow both? The group can have their speaker within whatever guidelines the school has (some schools ban foul language or other school-wide policies), and the other group is allowed to protest. Just like the real world where another company can do what it wants within the law, but I can also protest what and how they do it.

Protesting the speaker's ideas is always fine.

Protesting the speaker's existence is difficult to intellectually justify as a healthy exchange of ideas, including critique.

These days, what we see is a lot of the latter -- up to and including attempting to prevent attendance of the event -- and very little of the former.

I've seen cases on college campuses of obvious criminal activity, although usually minor, that is excused because they happen during a protest.