The problem is that when you put a file out into the world, you can't continuously update it with the latest advances in adversarial machine learning. It'll have a reprieve ranging from months to years, then be vulnerable forever more.

But there's an upper limit since the photos still need to be recognizable to people.

I think the better word is “trained”, not “robust”.

By making almost invisible (to humans) distortions to a picture, a previously well-trained algorithm is now facing a completely unknown problem.

That implies the amount of training is the only factor. It may be that new architectures or training methodologies could be developed to reject large swathes of these types of advesarial examples. Though of course, if adversarial training is used avdersarial examples will obviously exist at some level, and so the arms race continues.

But for every adversarial distortion algorithm, you’d in theory have to double your training set, wouldn’t you?

I'm not sure I follow the point about robust facial recognition being cheaper to break. Like others pointed out, I don't see how this could be a scalable tactic.