News for hackers. Stuff that matters.
Re:Estimating required GPU memory for serving LLMs by brianjking on Sunday November 19, 2023 @01:08PM (#38335646)
My issue is figuring out how to identify how many concurrent users you can support on average on a given GPU.

Understanding the vram to simply load the weights is easy enough. When you are allowing for something like content generation with varying lengths of input/output tokens, how do you even begin to identify the GPUs you need?

There are no comments on this comment yet.