I spent a few months recently on a tough ML project, and pretty much got my ass kicked the entire time. It seemed like every hurdle that I overcome was met with another as I turned up the "voltage". I came to regard every decision that I made (this includes all kinds of hyperparameters, but also design decisions) with extreme suspicion. I don't really think there is a convincing way around this: any kind of optimization has a context outside of which the optimization no longer makes sense. And so one tries to include this context, but this turns into a meta-optimization with a meta-context assumed, and so-on in infinite regress. I guess I am agreeing with the author: "if two algorithms achieve the same performance on a task, the one with less hyperparameter optimization is generally preferable."

It really seems like there should be more of a theory around these issues. Even a dreadfully abstract and/or terse VC-dimension-scariness level of a theory.

I keenly await the author's next post.