The danger also scales with the number of aggressive assholes wearing the same uniform. It is like any job where your coworkers don't do their job. Eventually you will be held responsible for their results. When the boss asks you about it you might be reluctant to point out your coworkers since you still have to work with them.