Re:Drink Me: (Ab)Using a LLM to Compress Text
by tednoob on Sunday May 05, 2024 @03:07AM
(#40262879)
Is this method used during training? Seems to me there could be a point to only backpropagate when the model is wrong?
[ Parent
| on HN ]
I mean this is implicit in back propagation, say, you need to store gradients anyway but if you get to a zero loss than you are just done.
Re:Drink Me: (Ab)Using a LLM to Compress Text
by leodriesch on Sunday May 05, 2024 @07:39AM
(#40264086)
The model is always wrong, since it predicts a propability distribution over all possible tokens, but the target has 100% possibility for one token and 0 for all others.