Re:Do Machine Learning Models Memorize or Generalize?
by jjk166 on Thursday August 10, 2023 @05:58PM
(#37082302)
Memorization is storing data. Generalization is developing the heuristics by which you compress stored data. To distill knowledge is to apply heuristics to lossily-compress a large amount of data to a much smaller amount of data from which you nevertheless can recover enough information to be useful in the future.
[ Parent
| on HN ]
Re:Do Machine Learning Models Memorize or Generalize?
by staunton on Thursday August 10, 2023 @08:06PM
(#37083396)
> Generalization is developing the heuristics by which you compress stored data
such that
> you nevertheless can recover enough information to be useful in the future.
I disagree (in case you meant to imply it) that compression implies generalization.
Re:Do Machine Learning Models Memorize or Generalize?
by jjk166 on Friday August 11, 2023 @10:19AM
(#37088970)
I did not mean to imply compression implies generalization, if anything the reverse. Compression is the act of cutting, generalization is the whetstone by which you may sharpen a blade, which is the compression heuristic. A more general heuristic is to compression what a sharper blade is to cutting.