News for hackers. Stuff that matters.
Re:Ask HN: Best way to deploy an LLM for 100s of user by K0IN on Friday July 05, 2024 @02:04PM (#40884773)
so for fast responses we usea a rtx4090 with vllm, but yeah it depends on your use case
Re:Ask HN: Best way to deploy an LLM for 100s of user by samspenc on Friday July 05, 2024 @04:40PM (#40885806)
Agreed, this works great for a single or small set of users (2-3), but OP is asking about self-hosting an open-source / open-weight LLM locally to serve a larger set of users (100s per post title). I suspect they may need a larger server farm with a lot of GPUs to handle the load.