Cloudflare's AI Platform: an inference layer designed for agents

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Thursday April 16, 2026 @04:30PM (#47799096)

So it's basically just openrouter with cloudflare argo networking? I feel like they could do some much more interesting stuff with their replicate acquisition. Application specific RL is getting so good but there's no good way to deploy these models in a scalable way. Even the providers like fireworks which claim to let you deploy LORAs in a scalable way can't do it. For now I literally have to host base load on my application on a rack of 3090s in my garage which seems silly but it saves me $1k a month.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by vladgur on Thursday April 16, 2026 @06:00PM (#47800073)

Curious which models are you able to run and how many 3090s do they require at scale?

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Thursday April 16, 2026 @06:20PM (#47800272)

4 3090s with nvlinks on each pair. Super fast inference on Moe models around 20-36b

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by embedding-shape on Friday April 17, 2026 @12:00PM (#47807376)

> Super fast inference

How fast is "super fast" exactly, and with what runtime+model+quant specifically? Curious to see how how 4x 3090s compare to 1x Pro 6000, could probably put together 4x 3090s for a fraction of the cost compared to the Pro 6000, but the times I've seen the tok/s in/out for multiple GPUs my heart always drops a little.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @12:43PM (#47807860)

I haven't benchmarked against a pro 6000, it's more that i have 4 3090s and i don't have a pro 6000.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by embedding-shape on Friday April 17, 2026 @01:00PM (#47808035)

Yes, that's why I'm asking you what exactly 4 3090s get in prompt-processing and generation, sorry if I was unclear.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @03:29PM (#47809656)

Maxes out around 4K tok/s output. Each pair of 3090s has its own instance of the model with parallelism across the nvlink bridge. Though nvlink is only 2x over pcie5

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by jonfromsf on Thursday April 16, 2026 @11:30PM (#47802204)

Gilfoyle? Is that you?

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @12:24AM (#47802437)

I think these gpus were actually used for bitcoin mining before I bought them

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by menno-dot-ai on Friday April 17, 2026 @12:54PM (#47807973)

It's Anton's grandson!

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by bryden_cruz on Friday April 17, 2026 @03:38AM (#47803437)

Running a rack of 3090s in your garage to avoid provider lock-in/costs is the most Hacker News thing. Out of curiosity, what are you doing for uptime/failover? If you are running production traffic to that garage rack, does your app just degrade gracefully if your home internet drops, or do you have a cloud fallback?

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @12:59PM (#47808022)

Yeah the model i'm running locally is just one of several models the app supports and it falls back to others if not available.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by ascorbic on Friday April 17, 2026 @07:39AM (#47804812)

The interesting part is that you can use the same API with Workers AI models (hosted at the edge) and proxied models (OpenRouter-style).

Disclaimer: I work at Cloudflare, but not on this.

[ Parent | on HN ]

Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @07:11PM (#47811584)

It's the same problem as fireworks, the only models supporting LORA are like year old dense models that perform horribly on most tasks. If you want to do anything close to relevant you still need to rent/own dedicated GPUs, which seems insane to me when vLLM fully support dynamic LORA loading.

[ Parent | on HN ]