Re:Bugs in LLM Training – Gradient Accumulation Fix
by danielhanchen on Friday October 18, 2024 @07:21PM
(#41884367)
Yes so making the sequences all the same ie through packing them into one still introduces issues since packing them into one has an unpadded token at the end due to the autoregressive nature of the training process.
[ Parent
| on HN ]
Re:Bugs in LLM Training – Gradient Accumulation Fix
by pama on Sunday October 20, 2024 @11:36AM
(#41896013)
Not sure what you mean. There is always an end to a batch. It doesn't have to be the end of a document entry, otherwise the model might get lazy and learn something related to the position of the text (i.e. look into the position encoding and call it a day).