Re:Bugs in LLM Training – Gradient Accumulation Fix
by danielhanchen on Friday October 18, 2024 @02:36PM
(#41882193)
Ye a complete change was necessary for now - HF had to isolate the cross entropy loss and make another class for it, and it had to be applied to all model archs.
[ Parent
| on HN ]
There are no comments on this comment yet.