News for hackers. Stuff that matters.

nstogner

Operating large language models (LLMs) at scale in real-world scenarios necessitates multiple backend replicas. The inter-replica load balancing algorithm greatly influences overall system performance by impacting the caching dynamics inherent in modern inference engines. In this study, we explore the application of the Consistent Hashing with Bounded Loads (CHWBL) algorithm in the domain of LLM inference. We simulate real-world load patterns on Kubernetes, the industry's most widely adopted deployment platform. Our results demonstrate a 95% reduction in Time To First Token (TTFT) and a 127% increase in overall throughput compared to the baseline strategy.

Show HN: SandboxAI – Run AI generated code in containers

https://github.com/substratusai/sandboxai

We created SandboxAI because we wanted to run AI generated code on our laptop without relying on a third party service. We also wanted something that would scale when we were ready to push to production. That's why we support docker for local execution and will soon be adding support for Kubernetes.Up next: {security features}++ such as egress rules, docs on how to run with hardened...

( Read More... | no comments | on HN )

Show HN: KubeAI – Open AI on Kubernetes

https://github.com/substratusai/kubeai

The open source AI ecosystem is progressing at a pace that is hard to keep up with. Our team is building KubeAI to make it easy to quickly operationalize OSS models and tools within Kubernetes. The goal is to serve as a drop-in replacement for proprietary platforms like OpenAI while also integrating well with existing cloud infrastructure (i.e. messaging systems, databases, etc). Design...

( Read More... | no comments | 3 points | on HN )
Re:Show HN: Kubectl Notebook on Friday August 25, 2023 @03:58PM (#37266323)
This issue is now fixed in v0.9.0 (check "kubectl notebook --version").
Re:Show HN: Kubectl Notebook on Thursday August 24, 2023 @01:31PM (#37251937)
Thanks! Kubernetes provides some solid building blocks for training LLMs. I believe that what is missing is a little tooling to link together those primitives into a streamlined ML DevX.