All words do, to some extent. "The greatest problem in communication is the illusion that it has happened at all." — George Bernard Shaw
I'm not sure it's much like the n-word beyond the intent of the utterer, though. I've been called "My n-word" before, by people who "get" to use that word, yet I'm white. I still wouldn't presume to use it back (for myriad reasons, most of which would be tangential, at best, to this discussion — yet more reason I don't find it so relevant).
My point has more to do with whether the term is being used constructively (or at least critically, in the sense of analysis, not of berating), versus whether it's being used as some kind of accusation of inherent badness. Because, basically every time the concept comes up, people leap into the conversation taking it that way.
Anyone who feels moved to can talk about "toxic [whatever]ity". How that conversation goes probably depends at least as much on what they mean by it as it does their audience.
The reason to not use the N-word is for me identical to toxic masculinity. Most recipients and people who happen to be near won't interpret it in any positive way. Both are unproductive for meaningful discourse and leads to a hostile environment. Since people who use those term is usually well aware of the reaction, it must be assumed that it is the speakers intention is to cause harm.
People who talk about "toxic masculinity", in the academic sense, shouldn't use that term, because other people will have feelings about it? People should adjust their speech to the sensitivities of — not just their, but any — audience? Is that legit the position here?
The answer was obvious and thus the pastry has a new name. Benign usage may take time to go away but if there is no negative intention then it is just easier to switch to other words. Even with traditional pastry people adjusted their speech because other people have feelings about it. Still people wrote quite a lot of political debate articles trying to argue that the whole thing were totally silly, irrational and people should just not get upset over the name of a pastry.
Back then I had similar line of thought as you and thought people should not have to adjust their speech because some unnamed and never defined person may get offended. It was after all just a pastry. It did not feel like something which anyone could legitimate be upset over. Now days I tend to see it more from a perspective of communication and if a word causes generally misunderstanding of the intentions of the speaker then it might just be best to use a other word.
It turns out, when you use those terms to describe traits, rather than as accusations of assholery or inherent awfulness, you can actually have a conversation, rather than shouting defensively at each other. I think it might be because as used in academic discourse, that's what they are: descriptions of traits.