I think it's because short sellers are seen as pessimists -- on some level, you're betting for a business to fail (or perform poorly), and regardless of whether or not you're right, that means you're betting on people losing jobs, dreams going up in smoke, etc. etc.

I don't subscribe to this narrative, but it's an easy one to spin when your stock is the one being short sold.