Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by jgrahamc on Wednesday September 27, 2023 @09:44AM
(#37674577)
TL;DR: GPUs all over the Cloudflare global network; working closely with Microsoft, Meta, Hugging Face, Databricks, NVIDIA; new Cloudflare-native vector database; inference embedded in Cloudflare Workers; native support for WebGPU. Live demo: https://ai.cloudflare.com/
[ Parent
| on HN ]
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by tebbers on Wednesday September 27, 2023 @10:39AM
(#37675320)
Hey John, great work on this! Just a headsup, small typo on that page under R2: "Build mutli-cloud training architectures with free egress."
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by jgrahamc on Thursday September 28, 2023 @05:18AM
(#37687168)
Thanks. Getting it fixed.
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by foggedb0nk on Wednesday September 27, 2023 @10:27AM
(#37675153)
Any chance you're looking for technical product folks to work on this? I actually worked on a very similar deployment internally at Livepeer (focus was on live video enhancements but also generalized edge compute)!
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by rita3ko on Wednesday September 27, 2023 @10:49AM
(#37675448)
we always are! email is rita at cloudflare dot com :)
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by foggedb0nk on Thursday September 28, 2023 @02:07PM
(#37693346)
thanks!
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by aprxi on Wednesday September 27, 2023 @10:17AM
(#37675016)
Is CodeLlama somewhere on the roadmap?
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by ushakov on Wednesday September 27, 2023 @09:52AM
(#37674665)
Do you actually run the inference in the worker? Or is it like what Fermyon does where they basically host the models for you and you get a SDK that is automatically connected to the function?
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by celso on Wednesday September 27, 2023 @10:01AM
(#37674790)
Unlike the first version of Constellation, Workers AI runs inference directly on GPUs that we are (quickly) installing in our global network.
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by ushakov on Wednesday September 27, 2023 @10:21AM
(#37675072)
But the code isn't running on the worker? It runs somewhere else on a GPU cluster?
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by jgrahamc on Wednesday September 27, 2023 @10:23AM
(#37675096)
It's a little like how Cloudflare Workers runs. You don't know which CPU it runs on, all you know is it's a CPU close to your end user. Same goes for this. We are rolling out GPUs everywhere across the globe and so Workers AI will just use a nearby GPU. Probably in the same machine as your workers, or maybe the same data center, or whatever other smart routing decision we make. What we are not doing is running a massive GPU cluster somewhere. This is all distributed and that's the power of owning your own network.
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by pseg134 on Wednesday September 27, 2023 @12:08PM
(#37676810)
Since they don’t seem to be able to give a simple answer: the inference does not run in the worker. It connects to external GPUs.
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by eastdakota on Wednesday September 27, 2023 @12:26PM
(#37677103)
I think the confusion is what is meant by "in the Worker." From a hardware perspective, the GPU may be in the same machine as the CPU that's powering the Worker. Or they may be across different machines in our network. We are not routing requests to some third party. And we will try to run the inference task as close as possible to who/whatever requested it. The whole idea of "serverless" is you shouldn't have to worry about what machine where runs whatever unless you're on the team building the scheduling and routing logic at Cloudflare.
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by thegagne on Wednesday September 27, 2023 @12:56PM
(#37677573)
I think his question is more about does the worker directly access the GPU and thus require js tooling to handle the GPU somehow (no), or does it make subrequests to a separate GPU service not running the worker runtime (yes).
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by claytonjy on Wednesday September 27, 2023 @11:08AM
(#37675751)
I see plans for more models via HF partnership, but can I or will I be able to run a custom fine-tuned version of a supported model?
Re:Workers AI: serverless GPU-powered inference on Cloudflar...
by celso on Wednesday September 27, 2023 @11:22AM
(#37675985)
On top of our hosted and supported catalog of models, and the deploy to CF partnerships like the HF one, you will also be able to bring your own custom model at some point in time.