Re:Tinybox of tinygrad by George Hotz is finally entering pr...
by andersa on Tuesday August 27, 2024 @10:21AM
(#41367857)
Most models actually can't be split nicely by 6. There's a reason nvidia builds nodes with 4 and 8 GPUs.
[ Parent
| on HN ]
Re:Tinybox of tinygrad by George Hotz is finally entering pr...
by StrangeDoctor on Tuesday August 27, 2024 @11:31AM
(#41368669)
I don't see why 6 is inherently worse than 4 or 8, not all of the layers are exactly equal or a power of 2 in count. 2^2, 2^3, vs 2^1*3^1 might give you more options.
The main issue I run into mainly is flops vs ram in any given card/model.
Re:Tinybox of tinygrad by George Hotz is finally entering pr...
by andersa on Tuesday August 27, 2024 @12:01PM
(#41368983)
Usually you want to split each layer to run with tensor parallelism, which works optimally if you can assign each kv head to a specific GPU. All currently popular models have a power of 2 number of kv heads.
Re:Tinybox of tinygrad by George Hotz is finally entering pr...
by StrangeDoctor on Tuesday August 27, 2024 @01:02PM
(#41369911)
interesting, thank you for the pointers.