Re:An overview of the theory of overparameterized machine le...
by IX-103 on Tuesday September 21, 2021 @02:56PM
(#28608487)
Memorization is only an issue if you allow it to be. If design the model with a "narrow" enough inner stage then that limits the level of detail (in terms of distinct representable values) passed to subsequent stages. This should give you an ML algorithm that consists of a fingerprint (approximates your hashing) stage followed by a classifier that works based on the fingerprint input (approximates a table lookup). Such an algorithm should not have such a problem with over-fitting was you describe.
[ Parent
| on HN ]
Re:An overview of the theory of overparameterized machine le...
by joe_the_user on Tuesday September 21, 2021 @08:52PM
(#28611924)
Memorization is only an issue if you allow it to be.
Sure, it's a potential problem that can appear in the process implementing a deep learning solution. It's not an insurmountable problem. But the fact that still appears seems like an indication the situation in deep learning is more complicated than "overparameterization is not a problem".