Re:Learning Theory from First Principles [pdf]
by ithinkso on Sunday March 03, 2024 @11:19AM
(#39581823)
One just assumes Occam's razor and you're good to go in a vast majority of cases
Re:Learning Theory from First Principles [pdf]
by bonoboTP on Sunday March 03, 2024 @11:29AM
(#39581886)
If you formalize Occam's "simplicity" as description length, then that depends on encoding. By assuming an encoding, you implicitly assume a distribution (according to entropy coding, eg Huffman coding) and hence inject an inductive bias.
I'm not saying that it's bad to assume one. The point is that it is an assumption. My bigger point is that the no free lunch theorem should only bother you as much as the induction problem bothers you. Which in practice means not at all.
But it seems like science is "doing fine" despite this problem. Similarly machine learning chugs along fine, because people use their experience and prior knowledge when designing the algorithms. These assumptions are also called inductive biases. They are biasing the learning towards certain patterns (like "things tend to be similar locally").