But this is not the only way you can deploy machine learning techniques. You could deploy unsupervised learning to first learn what visual features are common in such images, and then successively refine until you end up with a set of images containing very rare phenomena. That could give human evaluators then a considerable head start.

In fact, this idea seems so obvious that it must have been tried already…