Announcing GPT-NeoX-20B

Re:Announcing GPT-NeoX-20B by f38zf5vdt on Wednesday February 02, 2022 @12:08PM (#30180396)

Right on, they're closing in on "Open"AI's best models. Can this still be run on a GPU, or does it require a lot more VRAM?

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by stellaathena on Wednesday February 02, 2022 @12:24PM (#30180674)

It can be run on an A40 or A6000, as well as the largest A100s. But other than that, no.

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by bm-rf on Wednesday February 02, 2022 @04:47PM (#30184517)

You could use Microsoft's DeepSpeed to run the model for inference on multiple GPUS, see https://www.deepspeed.ai/tutorials/inference-tutorial/

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by djoldman on Wednesday February 02, 2022 @02:06PM (#30182337)

How much VRAM does it use during inference?

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by stellaathena on Wednesday February 02, 2022 @02:32PM (#30182769)

~40 GB with standard optimization. I suspect you can shrink it down more with some work, but it would require significant innovation to cram it into the next largest common chip size (24 GB, unless I’m misremembering)

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by komuher on Wednesday February 02, 2022 @04:46PM (#30184503)

Is 40GB already on float16?

[ Parent | on HN ]

Re:Announcing GPT-NeoX-20B by stellaathena on Friday February 04, 2022 @07:25PM (#30215572)

Yes

[ Parent | on HN ]