Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by metadat on Friday August 23, 2024 @07:54PM (#41334209)

How does 12 tokens/sec equate to satisfactorily severing thousands of end-customere?

I did enjoy the headline, and the register has formerly been an incredibly good news outlet. Prior to the founder passing away last week.

[ Parent | on HN ]

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by atherton33 on Friday August 23, 2024 @08:03PM (#41334271)

From the article, it's not 12 total, but 12 per user for 100 concurrent generations.

[ Parent | on HN ]

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by metadat on Friday August 23, 2024 @08:05PM (#41334275)

Thank you atherton friend.

[ Parent | on HN ]

Re:Benchmarks show even an old Nvidia RTX 3090 is enough to ... by washadjeffmad on Friday August 23, 2024 @09:03PM (#41334640)

Capacity doesn't equate to quality, but I could easily see an 8B finetune with exl2 at low context working for short, simple customer interactions, akin to oversubscribing a 1Gbit uplink for 100 customers at 50Mbps.

[ Parent | on HN ]