(Btw I keep running into you or your content the past couple months, thanks for all you do and your well thought out contributions -@jpohhhh)
Because the model isn't dynamic (it doesn't learn) it is stateless and can be scaled elastically.
How many users come with the same prompt?
I’ve had servers locked up in a cage for years without seeing them. And the cost for bandwidth has plummeted over the last two decades. (Not at AWS, lol)
I'm hosting a lot of stuff myself on my own hardware, so I do sympathize with this argument, but in a time>>money situation, going to the cloud makes a lot of sense.
300W per A100 * 8766 hours per year * $0.12 per kWh = $316 to power an A100 for a year
OpenAI doesn't have to make money right away. They can lose a small bit of money per API request in exchange for market share (preventing others from disrupting them).
As the cost of GPUs goes down, or they develop at ASIC or more efficient model, they can keep their pricing the same and then make money later.
They also likely can make money other ways like by allowing fine-tuning of the model or charging to let people use the model with sensitive data.
But he can't get away with it with all the competition in other companies coming on top of China, Russia and others also adopting AI development
> The company's investors pressured it to grow very fast to obtain first-mover advantage. This rapid growth was cited as one of the reasons for the downfall of the company.
IMO, selling at a loss to gain market share only makes sense if there are network effects that lead to a winner-takes-all situation. Of which there are some for ChatGPT (training data when people press the thumbs up/down buttons), but is that sufficient?
If engineers are getting into AI development through OpenAI, they're using tools and systems within the OpenAI ecosystem.
Daily on HN there's a post on some AI implementation faster than chatgpt. But my starting point is OpenAI. If you can capture the devs, especially at this stage, you get a force multiplier.
https://www.ftc.gov/advice-guidance/competition-guidance/gui...
Has that been happening? I guess there's been a bit of a dip after the crypto crash, but are prices staying significantly lower?
> or they develop at ASIC or more efficient model
This seems likely. Probably developing in partnership with Microsoft.
The cards that were used for mining have since crashed in terms of prices, but those were always gamer cards and very rarely Datacenter cards.
Nvidia needs to satisfy gamers, who individually can't spend more than a few $k on a processor. But they also have the server sector on lockdown due to CUDA. Seems they can easily make money in both places. Maybe those H100s aren't such a good deal...
If someone understands these dynamics better I'd be curious to learn!
By contract - you can’t sell 4090s in a data center. You’ll find a few shops skirting this, but nobody can get their hands on 100k 4090s without raising legal concerns.
Likewise, nvidia A100s have more than a few optimizations through nvlink which are only available on data center chips.
Lastly, per card memory matters a lot Nvidia has lead the market on the high end here.
(There are also some density advantages to the SMX form factor and the datacenter cards are passively cooled so you can integrate them into your big fan server or whatnot. But those differences are relatively small and certainly not on their own worth the price difference. It's mostly market segmentation.)
Maybe I'm just old but back in my day this would be called "dumping" or "anti-competitive" or "market distortion" or "unfair competition". Now it's just the standard way of doing things.
We can acknowledge that things were historically pretty horrible and strive to be better in the future.
- tolerate the current state of the chatbots
- tolerate the high per-query latency
- tolerate having all queries sent to OpenAI
- tolerate OpenAI [presumably] having 0 liability for ChatGPT just randomly hallucinating inappropriate nonsense
- be willing to pay a lot of money for the above
I'm kind of making an assumption on that last point, but I suspect this is going to end up being more small market business to business than mass market business to consumer. A lot of these constraints make it not really useable for many things. It's even somewhat suspect for the most obvious use case of search, not only because of latency but also because the provider needs to make more money per search after the bot than before. There's also the caching issue. Many potential uses are probably going to be more inclined to get the answers and cache them to reduce latency/costs/'failures' than endlessly pay per-use.
Anyhow, probably a lack of vision on my part. But I'd certainly like to know what I'm not seeing.
Videogames maybe?
https://www.youtube.com/watch?v=ejw6OI4_lJw
This prototype is certainly something to have an eye out for
Even e.g. to develop hardware such as planes and cars: https://assistedeverything.substack.com/p/todays-ai-sucks-at...
For a three year reservation that comes to over $96k/yr - to support one concurrent request.
e.g. Endpoint feeds a queue, queue fills a batch, batched results generate replies. You are simultaneously fulfilling many requests.
That said I don’t think anyone anyone outside of OpenAI knows what’s going on operationally. Same goes for VRAM usage, potential batch sizes, etc. This is all wild speculation. Same goes for whatever terms OpenAI is getting out of MS/Azure.
What isn’t wild speculation is that even with three year reserve pricing last gen A100x8 (H100 is shipping) will set you back $100k/yr - plus all of the usual cloud bandwidth, etc fees that would likely increase that by at least 10-20%.
We’re talking about their pricing and costs here. This gives a general idea what anyone trying to self host this would be up against - even if they could get the model.
This is 6 month of salary of one average developer's salary there. And BTW they are likely doing inference on 100s or 1000s of GPUs, not just 8.
8, 80k, or 800k GPUs depending on requirements and load - the point remains the same.
Does someone have a source for this?
(By the way, it is unknown how many parameters GPT-3.5 has, the foundation model which powers finetuned models like ChatGPT and text-davinci-003. GPT-3 had 175 billion parameters, but per the Hoffmann et al Chinchilla paper it wasn't trained compute efficiently, i.e. it had too many parameters relative to its amount of training data. It seems likely that GPT-3.5 was trained on more data with fewer parameters, similar to Chinchilla. GPT-3: 175B parameters, 300B tokens; Chinchilla: 70B parameters, 1.4T tokens.)
> For contexts and models with d_model > n_ctx/12, the context-dependent computational cost per token is a relatively small fraction of the total compute.
For GPT3, n_ctx is 4096 and d_model is 12228 >> 4096/12.
InstructGPT 2B outperformed gpt 3 175B, and chatgpt has a huge corpus of distilled prompt -> response data now.
I’m assuming most of these requests are being served from a much smaller model to justify the price.
OpenAI is fundamentally about training larger models, I doubt they want to be in the business of selling A100 capacity at cost when it could be used for training
I did some quick calculation. We know the number of floating point operations per token for inference is approximately twice the number of parameters(175B). Assuming they use 16 bit floating point, and have 50% of peak efficiency, A100 could do 300 trillion flop/s(peak 624[0]). 1 hour of A100 gives openAI $0.002/ktok * (300,000/175/2/1000)ktok/sec * 3600=$6.1 back. Public price per A100 is $2.25 for one year reservation.
[0]: https://www.nvidia.com/en-us/data-center/a100/
[1]: https://azure.microsoft.com/en-in/pricing/details/machine-le...