OpenAI ChatGPT: Optimizing Language Models for Dialogue

Re:OpenAI ChatGPT: Optimizing Language Models for Dialogue by doctoboggan on Wednesday November 30, 2022 @05:11PM (#33808576)

If you're opinion, what is the best model I can run on my M1 MBP with 64gb memory and 32 GPU cores?

[ Parent | on HN ]

Re:OpenAI ChatGPT: Optimizing Language Models for Dialogue by Vetch on Wednesday November 30, 2022 @06:30PM (#33809596)

I would like to say FlanT5 11B which is 45GB but my experience is if you're using huggingface the usual way, it can initially take up to 2x the memory of the model to load.

GPT-JT was released recently and seems interesting but I haven't tried it. If you're focused on scientific domain and want to do Open book Q/A, summarization, keyword extraction etc. Galactica 6B parameter version might be worth checking out.

[ Parent | on HN ]

Re:OpenAI ChatGPT: Optimizing Language Models for Dialogue by zaptrem on Wednesday November 30, 2022 @10:37PM (#33812040)

I'm able to run a 22b parameter GPT-Neo model on my 24gb 3090 and can fit a 30b parameter OPT model when combining my 3090 and 12gb 3080

[ Parent | on HN ]

Re:OpenAI ChatGPT: Optimizing Language Models for Dialogue by mdda on Thursday December 01, 2022 @06:32AM (#33814972)

Could you point to any resources online about how to do this? e.g. is this using 8-bit quantisation?

[ Parent | on HN ]