News for hackers. Stuff that matters.

A comment on Introducing ChatGPT and Whisper APIs. Parent comment

Re:Introducing ChatGPT and Whisper APIs by freeqaz on Wednesday March 01, 2023 @03:31PM (#34987885)
Per instance (worker serving an API request) it requires 8x GPUs. I believe they have thousands of these instances and they scale them up with load.

Because the model isn't dynamic (it doesn't learn) it is stateless and can be scaled elastically.

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by thewataccount on Wednesday March 01, 2023 @03:58PM (#34988250)
Ah okay, that makes a lot more sense thank you!
[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by pharmakom on Thursday March 02, 2023 @03:39AM (#34993451)
I expect some level of caching and even request bucketing by similarity is possible.

How many users come with the same prompt?

[ Parent | on HN ]
Re:Introducing ChatGPT and Whisper APIs by thewataccount on Thursday March 02, 2023 @08:07AM (#34995172)
In my experience running the same prompt always get's different results. Maybe they cache between different people but I'm not sure that'd be worth the cache space at that point? although 8x A100s is a lot to not have caching...
[ Parent | on HN ]