> If that's the case, then Uber has no right to avoid working with contractors simply because they continually refuse work.

Why? Uber has a right to do whatever they want for whatever reason or without any reason. Just like the other party in the contract, the driver, can stop accepting any rides and drive for lyft if he so wants. No reason required.