Re:Tinybox of tinygrad by George Hotz is finally entering pr...
by warkdarrior on Tuesday August 27, 2024 @02:09PM
(#41370802)
> 3B parameter model
That's tiny. Can it train/fine-tune 70B models?
That's tiny. Can it train/fine-tune 70B models?
Say you are training a 3B parameter model in BF16. That's 6GB of weights, as long as your step time is >=500ms you won't see a slowdown.