News for hackers. Stuff that matters.
Re:Bugs in LLM Training – Gradient Accumulation Fix by danielhanchen on Friday October 18, 2024 @02:36PM (#41882193)
Ye a complete change was necessary for now - HF had to isolate the cross entropy loss and make another class for it, and it had to be applied to all model archs.

There are no comments on this comment yet.