News for hackers. Stuff that matters.
Re:Drink Me: (Ab)Using a LLM to Compress Text by tednoob on Sunday May 05, 2024 @03:07AM (#40262879)
Is this method used during training? Seems to me there could be a point to only backpropagate when the model is wrong?
Re:Drink Me: (Ab)Using a LLM to Compress Text by zwaps on Sunday May 05, 2024 @03:57AM (#40263077)
I mean this is implicit in back propagation, say, you need to store gradients anyway but if you get to a zero loss than you are just done.
Re:Drink Me: (Ab)Using a LLM to Compress Text by leodriesch on Sunday May 05, 2024 @07:39AM (#40264086)
The model is always wrong, since it predicts a propability distribution over all possible tokens, but the target has 100% possibility for one token and 0 for all others.