That's a good rationalization indeed! It makes sense that you should be able to think and say "Y_Y is a jerk who deserves to die", but if you say it to my face then I might be reasonably upset to the point of considerable emotional harm, or if you say it from your pulpit it could reasonably be interpreted by one of your followers as an instruction to commit murder. The speech alone isn't the crime, but you can certainly commit different crimes purely by speaking (though the context is determinative).