I don't think it is negative as a concept. I agree that, in theory, it is best to always err on the side of more freedom, not less. In practice, however, it can become negative if you think people don't always act rationally. Which is why I don't know if the right to work is good policy; part of me feels like it takes advantage of humans short-term bias, and leads them to limit their own long-term rational best interests.
It's very much like the social media debate going on right now. Part of me just wants to say it's better to have a completely open marketplace and let the chips fall where they may. But another part of me feels like there is a very real danger this will hijack human psychological shortcomings in a way that is not good long term. Maybe the cliché that "people get the government they deserve" is the closest I can get to reconciling the two.
I'm unsure of the relevance to "freedom of association" as a negative concept, though. You are still free to associate with who you wish, just as corporations are free to not associate with you if you do not join a union.
If we are going to transform a negative right to not join a union into a positive one, there are many other things I'd like to turn into positive rights first.