News for hackers. Stuff that matters.
Re:Alternative clouds are booming as companies seek cheaper ... by zackangelo on Monday May 06, 2024 @02:28PM (#40277882)
Would like to take a moment to recommend fly.io for GPU workloads.

I've been building a prototype using them for the last couple of weeks and it's been great to use. I didn't have to jump through any hoops or apply for any quota adjustments to get started. And I especially appreciate how easy they make it to automatically scale your GPU instances to zero based on traffic.

Re:Alternative clouds are booming as companies seek cheaper ... by indigodaddy on Monday May 06, 2024 @03:06PM (#40278302)
Wow that is quite decent pricing actually. This makes me want to try to deploy something like llamafile/ollama or similar to fly.io+gpu for my personal on-demand llama/llm whims. (@simonw I’m looking at you— I feel like if you haven’t already done this on fly.io, that you’re probably thinking about it :) ) Seems private enough.. could throw some basic auth on top of it for me and trusted friends/family so I don’t get crazy bills. But privacy-wise with fly.io I think it’s good enough.

Only problem might be everytime it spins up to download the large model might be wasteful as far as getting charged for network/bw usage— wonder if it would be more cost-efficient to have persistent storage or just see how much time and bw it is to download on every cold start…

Re:Alternative clouds are booming as companies seek cheaper ... by Palmik on Monday May 06, 2024 @03:34PM (#40278622)
It's horrible horrible pricing! Their on demand price for A100 is what gets you H100 sxm in other places.
Re:Alternative clouds are booming as companies seek cheaper ... by dangrairo on Monday May 06, 2024 @05:04PM (#40279511)
I am curious what "other places" are you comparing it to.
Re:Alternative clouds are booming as companies seek cheaper ... by indigodaddy on Monday May 06, 2024 @06:58PM (#40280455)
Also we’d have to do apples to apples right? One cannot complain about fly.io gpu pricing and then point to say vast.ai or lambda labs as “evidence” of that. They aren’t at all the same type of service..
Re:Alternative clouds are booming as companies seek cheaper ... by benwaffle on Monday May 06, 2024 @04:51PM (#40279364)
We've got a one-liner for spinning up your own ollama UI. See https://github.com/fly-apps/ollama-open-webui
Re:Alternative clouds are booming as companies seek cheaper ... by indigodaddy on Monday May 06, 2024 @07:00PM (#40280463)
Oh snap thank you!
Re:Alternative clouds are booming as companies seek cheaper ... by nextworddev on Monday May 06, 2024 @03:02PM (#40278268)
Taking a step back, was it necessary to roll your own LLM (or whatever FM) API as opposed to using an off the shelf API
Re:Alternative clouds are booming as companies seek cheaper ... by zackangelo on Monday May 06, 2024 @08:27PM (#40281018)
Yes we’re running our own inference server because we do nonstandard things in the decoder loop (think logit masking, other types of sampling, etc).
Re:Alternative clouds are booming as companies seek cheaper ... by breakingcups on Monday May 06, 2024 @05:04PM (#40279506)
Hn's opinion of Fly seems to fluctuate a lot. First it was an HN darling, especially with all the high-quality blog posts they were putting out, but on the latest threads there's been a lot of complaining about platform stability and features.