There are some formal methods and rules of thumb for avoiding trouble in this regard. Take a look at bayesian and entropy-based maximization strategies. Hyperparameter optimization can be looked at using either strategy. Also take a look at https://en.wikipedia.org/wiki/Minimum_description_length

There's definitely a lot of theory, but it hasn't yet been turned into readily usable R libraries.