News for hackers. Stuff that matters.
Re:Bugs in LLM Training – Gradient Accumulation Fix by pama on Sunday October 20, 2024 @11:36AM (#41896013)
Not sure what you mean. There is always an end to a batch. It doesn't have to be the end of a document entry, otherwise the model might get lazy and learn something related to the position of the text (i.e. look into the position encoding and call it a day).

There are no comments on this comment yet.