Re:Bugs in LLM Training – Gradient Accumulation Fix
by ejddhbrbrrnrn on Friday October 18, 2024 @08:17AM
(#41878688)
Is this a general issue rather than unsloth specific. How wide is this problem? Sounds wild if it has been affecting everyones training.
[ Parent
| on HN ]
Re:Bugs in LLM Training – Gradient Accumulation Fix
by danielhanchen on Friday October 18, 2024 @02:32PM
(#41882151)
Unfortunately it's not an Unsloth issue but a general issue affecting nearly all trainers which use grad accum. We worked with Huggingface so their trainers should be fixed now though in the main branch