>What I disagree on is who should decide what hate speech is.

It is a risky business, but so are many others: food and drug regulation, road regulation, flight regulation etc. - the case must be made why freedom of speech is such a special right as opposed to any other action, such as selling food, driving cars, or building planes.

>Taking away the right for someone to speak their mind is too much power for one individual or group

We already do this globally, with copyright law, or less controversially, laws against threats, assault and child pornography. This at least shows that there is some scope, and unless that scope is a dogma, it should be open for questioning - either in broadening or narrowing.

Indeed, that's a power you'd have to be comfortable having in the hands of your opponents. Because at some point it will be and then they'll be the ones defining what speech is restricted.