News for hackers. Stuff that matters.

A comment on Groq runs Mixtral 8x7B-32k with 500 T/s. Parent comment

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by johntb86 on Monday February 19, 2024 @01:19PM (#39432929)
Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by pclmulqdq on Monday February 19, 2024 @05:02PM (#39435364)
You can control when those happen, too.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by Temporary_31337 on Monday February 19, 2024 @10:05PM (#39437588)
not without affecting performance though? If you delay refreshes, this lowers performance as far as I remember...
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by pclmulqdq on Tuesday February 20, 2024 @11:08AM (#39442973)
Control of all of this can come at a performance cost, but in the case of DRAM refreshes, it doesn't lower performance if you don't do them, it loses data. Nominally, you could do your refreshes closer together and as long as you know that the rows being refreshed will be idle and you have spare time on the bus, you're ok.
[ Parent | on HN ]