Re:Running a 180B parameter LLM on a single Apple M2 Ultra
by anurag6892 on Thursday September 07, 2023 @01:46PM
(#37422761)
this advantage is not specific to OpenAI right? Any big cloud provider like Amazon/Google can host these open LLM models.
Re:Running a 180B parameter LLM on a single Apple M2 Ultra
by smoldesu on Thursday September 07, 2023 @01:48PM
(#37422795)
It's not exclusive, no. At OpenAI's scale though, they can afford to purchase their own hardware like a big cloud provider can. It's likely that OpenAI was running Nvidia's DGX servers in production before AWS and GCP even got their unit quotes.
Is that not why OpenAI is winning right now? For free, you can have access to powerful AI on anything with a web browser. You don't need to wait for your SSD to load the model, page it into memory and swap your preexisting processes like it would on a local machine. You don't need to worry about the local battery drain, heat, memory constraints or hardware limitations. If you can read Hacker News, you can use AI.
Given the current performance of local models, I bet OpenAI is feeling pretty comfortable from where they're standing. Most people don't have mobile devices with enough RAM to load a 13b, 4-bit Llama quantization. Running a 180B model (much less a GPT-4 scale model) on consumer hardware is financially infeasible. Running it at-scale, in the cloud is pennies on the dollar.
I'm not fond of OpenAI in the slightest, but if you've followed the state of local models recently it's clear why they keep coming out ahead.