Re:Cerebras’ new monster AI chip adds 1.4T transistors
by etaioinshrdlu on Thursday April 22, 2021 @05:08PM
(#26908225)
It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications.
The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.
However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.
So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.
That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.
The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.
However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.
So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.
That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.