OpenAI closes its monster $10B funding round at $27B-29B valuation

Re:OpenAI closes its monster $10B funding round at $27B-29B ... by nullsense on Saturday April 29, 2023 @07:02AM (#35751820)

Going to need some insanely beefy GPU cluster at your house to run anything close to GPT-4.

The best I can run is either LLaMa 65B at 1 token per second (too slow) on my CPU or LLaMa 30B on my GPU at quite a fast ~30 tokens a second.

Nowhere near the usefulness of GPT-4.

[ Parent | on HN ]

Re:OpenAI closes its monster $10B funding round at $27B-29B ... by _hypx on Saturday April 29, 2023 @04:20PM (#35756762)

GPUs in the future will get more powerful. Especially if they are designed with LLMs in mind.

[ Parent | on HN ]

Re:OpenAI closes its monster $10B funding round at $27B-29B ... by nullsense on Saturday April 29, 2023 @06:58PM (#35758111)

I wonder where a critical threshold might lie? For example, right now if you only have 24Gb of VRAM, the best you can run locally at usable speeds are the 4-bit quantized LLaMa 30B models, which are only semi-usable. If you have 48Gb however, you can run the 4-bit quantized LLaMa 65B, which is much better.

I assume there will be advances that basically make the context window practically infinite. That ought to make the lower parameter count models much more powerful on its own. I also assume they will become more efficient to run through sparsity/pruning, though I'm a total novice on this topic.

I wonder how many generations of hardware are we talking? One? Two? Or three? It feels like it's potentially within that range.

[ Parent | on HN ]