Ask HN: How to deploy LLMs to serve a million users?

Re:Ask HN: How to deploy LLMs to serve a million users? by FrenchDevRemote on Tuesday April 11, 2023 @04:58PM (#35531456)

i guess you'd need a good relationship with a cloud provider, because I'm not even sure a regular customer could rent that many GPUs through normal means, or just make your own servers/datacenters

then I guess some basic decent load balancing/queuing on your inference endpoints

[ Parent | on HN ]

Re:Ask HN: How to deploy LLMs to serve a million users? by dbish on Wednesday April 12, 2023 @02:09AM (#35535623)

Exactly. All the big ones have partnerships/allegiances or own their cloud already, look at OpenAI and Microsoft or Google Bard on top of GCP, and similarly Stability seems to have a strategic partnership with AWS. You won't run at their scale without something like that, especially with GPU shortages we're seeing.

[ Parent | on HN ]