One way to fix this is a social safety net. If things like healthcare and the ability to pay rent are not dependent on employment, the worker's effective right to free speech and the employer's right to freely associate are both preserved.