Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool
by trifurcate on Tuesday February 14, 2023 @01:04PM
(#34793308)
I am running GPT-J-6B inference on 13GB of VRAM. FP16 is a must at this scale. I've also loaded GPT-NeoX-20B using int8 quantization onto a single RTX 3090 before (24GB VRAM) however inference latency was quite high.
[ Parent
| on HN ]
There are no comments on this comment yet.