News for hackers. Stuff that matters.

A comment on An embarrassingly simple approach to recover unlearned knowledge for LLMs. Parent comment

Re:An embarrassingly simple approach to recover unlearned kn... by jebarker on Monday November 04, 2024 @08:44AM (#42041456)
More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn't accessible at the lower precision
[ Parent | on HN ]
Re:An embarrassingly simple approach to recover unlearned kn... by spencerchubb on Monday November 04, 2024 @08:49AM (#42041487)
Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
[ Parent | on HN ]
Re:An embarrassingly simple approach to recover unlearned kn... by jebarker on Monday November 04, 2024 @08:57AM (#42041555)
I don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products
[ Parent | on HN ]
Re:An embarrassingly simple approach to recover unlearned kn... by fjdjshsh on Tuesday November 05, 2024 @11:32AM (#42052884)
If the model wasn't optimized with a parameter = 3.08 (that is, the optimization figured 3.08 increased the objective function), why would it be better than parameter = 3.1 ?

(This applies to functions of parameters too)

[ Parent | on HN ]