News for hackers. Stuff that matters.
Re:Bugs in LLM Training – Gradient Accumulation Fix by oergiR on Friday October 18, 2024 @04:57AM (#41877547)
That makes it sound like it’s a choice, which it isn’t really. The way to look at it is from a probabilistic perspective: with the fix, you maximise the probability of the data. Without the fix, you fairly arbitrarily raise some probabilities to a power greater than one, and some to a power less than one.
Re:Bugs in LLM Training – Gradient Accumulation Fix by danielhanchen on Friday October 18, 2024 @02:38PM (#41882219)
Yes exactly- mathematically it was incorrect to begin with.