Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

Re:Powering cost-efficient AI inference at scale with Cloud ... by amrutha_ on Tuesday November 28, 2023 @08:44PM (#38454379)

Fantastic blog! In your post, you highlighted the use of Cloud TPU v5e with GKE for AI inference. How does this setup maintain high performance while managing costs, especially in high-demand scenarios like real-time data processing or live interactions?

[ Parent | on HN ]

Re:Powering cost-efficient AI inference at scale with Cloud ... by bobbypage on Tuesday November 28, 2023 @09:04PM (#38454530)

Thank you! Regarding performance, TPU v5e which was benchmarked in the MLPerf results and showed impressive performance/dollar, see https://cloud.google.com/blog/products/compute/performance-p... which has more details. Combined with k8s & GKE, TPU v5e workloads can leverage auto-scaling, for example setting up autoscaling based on traffic, so workloads can scale down rapidly when not in use, increasing cost efficiency.

[ Parent | on HN ]

Re:Powering cost-efficient AI inference at scale with Cloud ... by ametrau on Wednesday November 29, 2023 @10:57AM (#38461041)

Can I use LLMs to make more convincing sock puppet questions?

[ Parent | on HN ]