News for hackers. Stuff that matters.

A comment on Cloudflare's AI Platform: an inference layer designed for agents. Parent comment

Re:Cloudflare's AI Platform: an inference layer designed for... by embedding-shape on Friday April 17, 2026 @12:00PM (#47807376)
> Super fast inference

How fast is "super fast" exactly, and with what runtime+model+quant specifically? Curious to see how how 4x 3090s compare to 1x Pro 6000, could probably put together 4x 3090s for a fraction of the cost compared to the Pro 6000, but the times I've seen the tok/s in/out for multiple GPUs my heart always drops a little.

[ Parent | on HN ]
Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @12:43PM (#47807860)
I haven't benchmarked against a pro 6000, it's more that i have 4 3090s and i don't have a pro 6000.
[ Parent | on HN ]
Re:Cloudflare's AI Platform: an inference layer designed for... by embedding-shape on Friday April 17, 2026 @01:00PM (#47808035)
Yes, that's why I'm asking you what exactly 4 3090s get in prompt-processing and generation, sorry if I was unclear.
[ Parent | on HN ]
Re:Cloudflare's AI Platform: an inference layer designed for... by mips_avatar on Friday April 17, 2026 @03:29PM (#47809656)
Maxes out around 4K tok/s output. Each pair of 3090s has its own instance of the model with parallelism across the nvlink bridge. Though nvlink is only 2x over pcie5
[ Parent | on HN ]