Volumes have been written about this topic (principle of least assumption, parsimonious representation, Occam's razor, etc.). It underlies all statistical learning theory and information theory of which science is a special case.