GPT4 is 8 x 220B params = 1.7T params

Re:GPT4 is 8 x 220B params = 1.7T params by mark_l_watson on Wednesday June 21, 2023 @09:17AM (#36418078)

There are huge differences. I used OpenAI’s APIs in my last book on LangChain and LlamaIndex, and GPT-3.5 and GPT-4 are good enough right now to support building applications for years (although I look forward to improvements).

I am writing a new book Safe For Humans AI in which I am constraining myself to using open models that can be run on a high end PC or a leased GPU server. Yesterday I was exploring what I could do with T5-flan-XXL, and it is useful, but not as effective tool as the closed OpenAI models. I will do the same with Orca this week.

[ Parent | on HN ]

Re:GPT4 is 8 x 220B params = 1.7T params by espadrine on Wednesday June 21, 2023 @09:57AM (#36418531)

I think a lot of people see a larger gap between open-source models and GPT-4 than is present because they test out the 7B models which fit on their machine. Models like T5-FLAN-XXL are very far below the quality one can expect from the best in open-source, and barely usable for CoT or tool use.

Especially for LangChain, I recommend using ≥33B models like Guanaco or WizardLM. Guanaco-65B honestly feels on-par with ChatGPT-3.5. (To be clear, there is a large gap with GPT-4 though.) It is a costly test, although GPTQ (for instance on exllama) help make it affordable.

I haven’t tried Orca since they haven’t released the weights yet, but it doesn’t seem like they have a 33B version.

[ Parent | on HN ]

Re:GPT4 is 8 x 220B params = 1.7T params by mark_l_watson on Wednesday June 21, 2023 @10:27AM (#36418881)

Thanks for your advice! I am trying to write the book examples using a A6000 (48G video memory), but I may have to go higher end. The hourly lease charge on a A6000 is very inexpensive and I wanted the examples to be "almost free" to run. I will see what is required for Guanaco-65B.

[ Parent | on HN ]