News for hackers. Stuff that matters.
Re:Tinybox of tinygrad by George Hotz is finally entering pr... by andersa on Tuesday August 27, 2024 @12:01PM (#41368983)
Usually you want to split each layer to run with tensor parallelism, which works optimally if you can assign each kv head to a specific GPU. All currently popular models have a power of 2 number of kv heads.
Re:Tinybox of tinygrad by George Hotz is finally entering pr... by StrangeDoctor on Tuesday August 27, 2024 @01:02PM (#41369911)
interesting, thank you for the pointers.