The "not strictly" only adds to the insult. You have group A, people who would be judged clinically insane (or borderline, which is more of an emotional issue). Let's call them true crazy. True crazy is bad. You have group B, much larger, people with inappropriate behavior that probably isn't going to lead to a psychiatric diagnosis, perhaps rather they just have a lack of self-control or good judgement. Let's call them false crazy. False crazy is a slur because of what it says, that your behavior is bad, bad like true crazy. And so the thing is, every time you use it, what you're also saying is that true crazy is worse than this.

The "acceptable use" that you're claiming is basically the same as calling someone or something gay, or someone a fggt (sorry, HN probably bans comments for using that word), even though the person and/or behavior in question has nothing to do with homosexuality. I hope this clarifies things a bit.