Re:Chiplet ASIC supercomputers for LLMs like GPT-4
by yvn1uo on Wednesday July 12, 2023 @06:11PM
(#36702363)
One possible explanation is that they hit the teraflops number during the prefill stage, where you can process all tokens at once, and are generally more operationally intensive, so you can use more compute. Utilization usually drops during the token generation stage. The utilization of the TPU during token generation is 3% when batch size 16. (https://arxiv.org/pdf/2211.05102.pdf, Table on the last page).
[ Parent
| on HN ]
Re:Chiplet ASIC supercomputers for LLMs like GPT-4
by why_only_15 on Thursday July 13, 2023 @01:11AM
(#36705242)
I mean maybe? This seems unlikely. I agree that decode is much more expensive and tok/s depends a lot on what your ratio of decode tokens to prefill tokens is.
This table was very helpful by the way, I didn't see that before. To me it clearly shows that 211 tok/s/A100 is very plausible and in fact kind of a poor showing because if you look at table D.4 and specifically the results for BS=256 PP3/TP8 they achieve ~150 tok/s/A100 on a model that's 3x larger than GPT-3.