If you think of the pursuit of new physics as an optimisation problem then the bias toward "beauty" is a guard against overfitting. Additionally, part of being "beautiful" to a physicist means that it gives insight into the "why" of the problem which helps in the pursuit of the next bit of new physics.