Cerebras Wafer Scale Engine Gen2 7nm 2.6T Transistors

Re:Cerebras Wafer Scale Engine Gen2 7nm 2.6T Transistors by modeless on Wednesday August 19, 2020 @02:18AM (#24207693)

Yeah, but it's not enough. The whole chip doesn't have enough SRAM to hold a tenth of GPT-3, let alone GPT-4. These are the kinds of models you would hope a chip like this would help with, but as soon as the model gets too big to fit in the SRAM then the advantage goes way down. And I think the models of the future will continue to get much, much bigger.

[ Parent | on HN ]

Re:Cerebras Wafer Scale Engine Gen2 7nm 2.6T Transistors by petra on Wednesday August 19, 2020 @02:37AM (#24207788)

There are a few companies doing analog compute-in-memory for ML. Usually on flash processes.

Maybe cerebras could fit well for that.

As for the models of the future getting much bigger - is the GPT family representative ? Are we seeing the same growth is in other domains ? And isn't there a big niche for smaller than GPT models ?

[ Parent | on HN ]