george_123
Re:Continuous batch enables 23x throughput in LLM inference and reduce...
on Tuesday August 15, 2023 @03:19PM (#37138107)
1) when the blog was released, TGI didn’t support paged attention, 2) many people don’t even know about TGI to reduce inference costs.
Re:Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML
on Monday June 26, 2023 @01:03PM (#36482524)
A lot of AWS services (especially SageMaker) aren’t very good in customer experience. People buy them for nominal capabilities and AWS core bread and butter — short-term and long-term reliability.
Most of these startups (AI and others) have to offer a compelling product before even being notable.
Besides, AWS top level doesn’t care if you use sagemaker or not. There’s a premium but if you’re still using EC2 via another startup, they’re still capturing lions share of value.
How continuous batching improves LLM inference throughput 23x
Re:vLLM: 24x faster LLM serving than HuggingFace Transformers
on Tuesday June 20, 2023 @05:58PM (#36411268)
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
Re:Ant Group – scaling to 1.37M QPS on Ray
on Tuesday December 13, 2022 @02:53PM (#33974755)
This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.
(resubmission, URL redirected to /Engineering last time)
Ant Group – scaling to 1.37M QPS on Ray
Re:
on Tuesday December 13, 2022 @01:44PM (#33973728)
This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.