Some companies have significantly more power than some governments in existence, so why shouldn't they have to act as if they were in fact a government. So, for example, if a mom and pop wants to fire someone because they don't like what they're saying, then more power to them. They don't have very much power, so their ability to defacto infringe on rights is limited. On the other hand, if Walmart doesn't like what I'm saying, then they have to give me 1st amendment considerations because of their vast amount of power.
The other side of this is that if you're running a small business and don't have the power to deal with employees who are acting in bad faith, then your business might actually tank. However, Walmart has got the resources to deal with employees who aren't just exercising their 1st amendment rights, but who are actively malicious and hurtful. Walmart can rotate the schedules or transfer people to another store or create a special position where the problem is sufficiently isolated. And Walmart can do all this without breaking a sweat.
This of course doesn't fix the "what happens if everyone hates you in a small rural area" problem. Theoretically all the entities in a rural area are going to have sufficiently low amounts of power AND proving collusion is going to be basically impossible (not to mention do you really want to force two people to hang out with a third just because there's only three people in an area and technically the two people have a high relative power base).
If I say something you don't like, then we all generally agree you can stop associating with me.
If I say something the government does not like, then we all generally agree they cannot do something to me as a consequence - as the inescapable authority means it would be a punishment.
Those are the two extremes. The middle is a grey area based on power dynamics. For example, someone says something their employer does not like. If they are let go but can simply find another job, then we can argue that employer is merely exercising free association. If they're from a more rural area and get blackballed at every employer in the area, then that effectively constitutes a restraint on speech.