Re:Ask HN: Best way to deploy an LLM for 100s of user
by K0IN on Friday July 05, 2024 @02:04PM
(#40884773)
so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case
[ Parent
| on HN ]
Re:Ask HN: Best way to deploy an LLM for 100s of user
by samspenc on Friday July 05, 2024 @04:40PM
(#40885806)
Agreed, this works great for a single or small set of users (2-3), but OP is asking about self-hosting an open-source / open-weight LLM locally to serve a larger set of users (100s per post title). I suspect they may need a larger server farm with a lot of GPUs to handle the load.