Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by Havoc on Sunday August 25, 2024 @08:45PM (#41352883)

Bought a 3090 because they are good value for this, but this logic is frankly a little ridiculous:

> Since only a small fraction of users are likely to be making requests at any given moment

So what if 5 out of the thousands happen to coincide?

[ Parent | on HN ]

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by kristoo on Monday August 26, 2024 @05:46AM (#41355513)

The benchmark result is for 100 concurrent users

[ Parent | on HN ]