News for hackers. Stuff that matters.
Re:Bugs in LLM Training – Gradient Accumulation Fix by ejddhbrbrrnrn on Friday October 18, 2024 @08:17AM (#41878688)
Is this a general issue rather than unsloth specific. How wide is this problem? Sounds wild if it has been affecting everyones training.
Re:Bugs in LLM Training – Gradient Accumulation Fix by danielhanchen on Friday October 18, 2024 @02:32PM (#41882151)
Unfortunately it's not an Unsloth issue but a general issue affecting nearly all trainers which use grad accum. We worked with Huggingface so their trainers should be fixed now though in the main branch