News for hackers. Stuff that matters.

A comment on Introducing ChatGPT and Whisper APIs. Parent comment

Re:Introducing ChatGPT and Whisper APIs by kkielhofner on Wednesday March 01, 2023 @06:33PM (#34989901)
Hopefully they’re doing plenty of batching - you don’t even need to roll your own as you’re describing. Inference servers like Triton will dynamically batch requests with SLA params for max response time (for example).

That said I don’t think anyone anyone outside of OpenAI knows what’s going on operationally. Same goes for VRAM usage, potential batch sizes, etc. This is all wild speculation. Same goes for whatever terms OpenAI is getting out of MS/Azure.

What isn’t wild speculation is that even with three year reserve pricing last gen A100x8 (H100 is shipping) will set you back $100k/yr - plus all of the usual cloud bandwidth, etc fees that would likely increase that by at least 10-20%.

We’re talking about their pricing and costs here. This gives a general idea what anyone trying to self host this would be up against - even if they could get the model.

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by vishal0123 on Wednesday March 01, 2023 @08:04PM (#34990613)
> will set you back $100k/yr

This is 6 month of salary of one average developer's salary there. And BTW they are likely doing inference on 100s or 1000s of GPUs, not just 8.

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by kkielhofner on Thursday March 02, 2023 @06:58PM (#35003822)
Yes and a devops engineer to manage an even moderately complex cloud deployment is an average of an extra $150k/yr. I don't know where this "cloud labor skill, knowledge, experience, and time is free" thinking comes from.

8, 80k, or 800k GPUs depending on requirements and load - the point remains the same.

[ Parent | on HN ]