The troubles people fear come from where the people creating and implementing law become increasingly disconnected from the enforcement, and removing people from enforcement removes a feedback from 'the ground'. A person can refuse to enforce what it perceives to be an unjust law. A union can band together and refuse to do certain things.

That can happen with robots as well, but the requirements for cooperation are lower. Fewer bad actors are required to achieve a super-bad goal. Imagine the entire police force controlled by one person. The entire army. Would that be good?

If you answer is no, or "maybe, it depends" then you also appreciate the unease that people feel about increased concentration of power that could come from automation.