You are never an asshole for telling the truth.

I've dealt with this before, it is a form of gas-lighting. Some people are good at making everyone else into a bully when THEY are the actual bully. Like the kid who keeps splashing you in a pool, but runs off and cries and tells when you splash them back.

Standing up for yourself sometimes makes you look/feel like an asshole. That doesn't mean you are wrong or that you shouldn't do it.

edit: Check out the book "Radical Candor" if you regularly struggle with expressing negative feedback