It doesn't matter if you interact only with people you follow, given that anyone with a bit of an audience gets plenty of hateful replies.
People have been putting up with being called "literally Hitler" and "literal nazis" (to somewhat match your example) almost since Twitter's inception, and still do, to a lesser degree, though, because levelling the playing field has seemingly made many of those people leave the site. Taking a look around was just short of unbearable because of the rampant censorship and repressive tactics that skewed the site very far into wokeness territory, up until Elon Musk took over. Yet people who didn't abide by it still put up with it, fought it, or downright ignored it.
Of course, I'm being downvoted for this, but (You) can't escape reality, folks. It was a shithole and we all endured it; now that it's free from the clutches of wokeness, it suddenly becomes unbearably toxic. Great. Find another echo chamber where you can mold reality to fit your abject fantasies. This time you won't be able to force them on us, though.
No ads?
There's a concept of "background radiation" expressed in social spaces. Dealing with a constant barrage of people who hate you or your existence[1] is tiring.
[1] Or perhaps they claim they don't hate you in particular, just, you know, anyone like you who they don't know in particular.
On Twitter, I have a make an active effort to not click on the "For you" tab because I'll be bombarded with posts about Trump or "woke" games, which I simply don't want to see. On Bluesky, I can actually discover new content without having to think about my mental health when using the "Discover" feed.
Do you have to defend, or can you just ignore. I assume those statements are still being made, even if you don’t read them. So why not just ignore and move on?
FWIW, Twitter (not saying Twitter is the best or only site) allows you to have a feed of only people you follow. That probably approximates going to another site of only people who share your core beliefs.