Ok, I understand what you are saying. Essentially, you would have liked different type of perturbations tested to get a baseline effect of how much a random perturbation can get people to agree. They did do this in what they call Experiment 3. Their control perturbation is simply the adversarial perturbation but flipped left/right. They claim this was to preserve perturbation statistics. Still, I think baselining against a control perturbation is not the point of the study. My takeaway of the study is that they give a constructive way of influencing human perception of images.

I understand that the concern could be something like "random perturbations of cat images make every image simultaneously less cat-like and hence more like anything else". My opinion is that Experiment 4 (making an image more cat-like or truck-like) covers concerns of this nature. Even if there are two random perturbations where one makes an image cat-like and the other more truck-like, it is completely arbitrary whether you label the perturbation as cat-like or truck-like (since they are randomly generated). That means, even if you measure a difference (and even if two random perturbations have larger differences than this construction!), you cannot control the direction. This method gives you a way to control it. Personally, I don't think this study is about measuring the influence over some baseline. It's about showing that you can indeed choose the direction of influence.