DiederikVink
When looking at a variety of workloads, we realized that prioritizing finishing a query (priotizing decodes) lead to underutilization of the GPU. We noticed there tended to not be enough requests that are concurrently running (because prefill wasn't prioritized) to meaningfully utilize the memory bandwidth with available decodes. This lead to a system that was unfortunately neither compute nor memory bound.
By running mixed batches that prioritize prefills we still compute some decode tokens in our spare capacity, but ensure compute is as saturated as possible. This additionally leads to a buildup of decodes, so that when we are primarily computing decode we're pushing our memory bandwidth as much as we can.
Of course there is still plenty of improvements that can be made on this front. Finding a dynamic balance between prefill and decode that allows us to have both the memory bandwidth and compute being pushed to their limits is the goal from a scheduling perspective. There are a whole host of factors such as the model architecture, input-token:output-token ratio, underlying hardware, KV-cache allocation (and many more) that all play into the pressure placed on memory and compute, so there's definitely still exploration to be done!
If a fully self-serve system is something you would like to see, we would love to hear more!
2. Could you please elaborate on the 50% cheaper option? If you're referring to the line on our website, that is due to our efficiency at scale. This efficiency benefit allows us to provide the models at the price that we do without implementing rate limits to manage our costs. Additionally, this 50% more efficient GPU utilization also benefits anyone looking to use our infrastructure for on-prem solutions.
Compared to the replicate/modal solutions our big focus is to ensure you don't experience rate limits. We want to ensure you get a good quality of service no matter what.
When it comes to requesting and running specific models, we won't ask you to pay extra just because there's lower demand for that specific model (which it sounds like other providers are doing). We manage scaling up and down instances for the models on your behalf to make sure you get good performance at a fair price point, so you don't have to worry about making the costs work.
We're currently working on providing a more extensive interface to show users a variety of performance metrics of the models they're running. Having efficiency metrics would be a great addition.
I think additionally an important facet of these tests would be providing clarity on the details of the tests to make them reproducible. I find that sometimes reported stats don't quite translate to real-world experiences. It can feel like results are presented using the workloads that look best on a system, so a standardized/reproducible approach would be best.
We're always keen to chat to as many users/experts/academics/enthusiasts as possible. Please feel free to reach me at [email protected] and we can set up a time to meet!
Although we're currently only supporting text models, we do definitely have image and video generation models in our roadmap as these are very compute intensive models meaning they would benefit greatly from optimizations. We'd love to hear more about any specific models you're hoping to run! Please feel free to message us with further details ([email protected]).
If I had to hazard a guess, it would be that their system architecture (# of chips and chip architecture itself) might not be designed for a high concurrency situation.
We've put a wrong hyperlink on the website, but we've fixed that now, thanks for letting us know.
Regarding us being able to reliably host models versus setting up a website largely comes down to our technical background. All of us are hardware engineers so our front-end capabilities are not our strong suit :). But our experience as hardware engineers makes us confident in hosting the models themselves. Both 8B and 70B models, if cached, do actually load in exactly 40s, but please feel free to try out the system and see for yourself!