GPT4 is 8 x 220B params = 1.7T params

Re:GPT4 is 8 x 220B params = 1.7T params by espadrine on Wednesday June 21, 2023 @09:57AM (#36418531)

I think a lot of people see a larger gap between open-source models and GPT-4 than is present because they test out the 7B models which fit on their machine. Models like T5-FLAN-XXL are very far below the quality one can expect from the best in open-source, and barely usable for CoT or tool use.

Especially for LangChain, I recommend using ≥33B models like Guanaco or WizardLM. Guanaco-65B honestly feels on-par with ChatGPT-3.5. (To be clear, there is a large gap with GPT-4 though.) It is a costly test, although GPTQ (for instance on exllama) help make it affordable.

I haven’t tried Orca since they haven’t released the weights yet, but it doesn’t seem like they have a 33B version.

[ Parent | on HN ]

Re:GPT4 is 8 x 220B params = 1.7T params by mark_l_watson on Wednesday June 21, 2023 @10:27AM (#36418881)

Thanks for your advice! I am trying to write the book examples using a A6000 (48G video memory), but I may have to go higher end. The hourly lease charge on a A6000 is very inexpensive and I wanted the examples to be "almost free" to run. I will see what is required for Guanaco-65B.

[ Parent | on HN ]