Hi there! Last month at NeurIPS (an ML conference), I read an interesting paper "Human Expertise in Algorithmic Prediction" that describes a framework for determining where ML models are outperformed by human experts. I found the authors' work to be very interesting. Below, I explore their framework further and extend it to multiclass classification. My results are pretty surprising, showing that a group of modern model architectures have trouble with dogs and cats in CIFAR-10.

GitHub Link: https://github.com/sunildkumar/model_indistinguishability

Paper Link: https://arxiv.org/abs/2402.00793