Thanks for the reply! I'm actually working in a different domain but it seems to have a lot in common with yours - lots of unlabelled data, images that have nothing in common with Imagenet, in that they are all essentially of the same thing and we are looking for variations or features. We found that self-supervised pre-training (with various contrastive models) underperformed vs. starting with weights trained on Imagenet.

So a model that has been pretrained on something else, with enough variability to work as a feature extractor, but closer to the problem framing I mention would be of interest.

For state of the art computer vision stuff, most of the benchmarks use imagenet or similar datasets. But unfortunately I'm coming around to the realisation that those datasets are not representative of most real world problems (except general purpose scene / object recognition). So it becomes very challenging to pick out a potential technique to apply, and hope it transfers.