Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by pclmulqdq on Monday February 19, 2024 @12:27PM
(#39432309)
I don't know about HBM specifically, but DDR and GDDR at a protocol level are both deterministic. It's the memory controller doing a bunch of reordering that makes them non-deterministic. Presumably, if that is the reason you don't like DRAM, you could build your compiler to be memory-layout aware and have the memory controller issue commands without reordering.
[ Parent
| on HN ]
That could be possible. It's out of my area of expertise so I can't say for sure. My understanding was HBM forces on you specific access patterns and non-deterministic delays. Our compiler already deals with many other forms of resource-aware scheduling so it could take into account DRAM refreshes easily, so I feel like there must be something else that makes SRAM more suitable in our case. I'll have to leave that to someone more knowledgeable to explain though ...
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by johntb86 on Monday February 19, 2024 @01:19PM
(#39432929)
Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by pclmulqdq on Monday February 19, 2024 @05:02PM
(#39435364)
You can control when those happen, too.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by Temporary_31337 on Monday February 19, 2024 @10:05PM
(#39437588)
not without affecting performance though? If you delay refreshes, this lowers performance as far as I remember...
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by pclmulqdq on Tuesday February 20, 2024 @11:08AM
(#39442973)
Control of all of this can come at a performance cost, but in the case of DRAM refreshes, it doesn't lower performance if you don't do them, it loses data. Nominally, you could do your refreshes closer together and as long as you know that the rows being refreshed will be idle and you have spare time on the bus, you're ok.