Re:Bugs in LLM Training – Gradient Accumulation Fix
by oergiR on Friday October 18, 2024 @04:57AM
(#41877547)
That makes it sound like it’s a choice, which it isn’t really. The way to look at it is from a probabilistic perspective: with the fix, you maximise the probability of the data. Without the fix, you fairly arbitrarily raise some probabilities to a power greater than one, and some to a power less than one.
[ Parent
| on HN ]
Re:Bugs in LLM Training – Gradient Accumulation Fix
by danielhanchen on Friday October 18, 2024 @02:38PM
(#41882219)
Yes exactly- mathematically it was incorrect to begin with.