News for hackers. Stuff that matters.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by isoprophlex on Wednesday June 04, 2025 @05:57AM (#44179006)
All the cruft of a big cloud provider, AND the joy of uncapped yolo billing that has the potential to drain your creditcard overnight. No thanks, I'll personally stick with Modal and vast.ai
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by montebicyclelo on Wednesday June 04, 2025 @06:53AM (#44179337)
Not providing a cap on spending is a major flaw of GCP for individuals / small projects.

With Cloud Run, AFAIK, spending can effectively be capped by: limiting concurrency, plus limiting the max number of instances it can scale to. (But this is not as good as GCP having a proper cap.)

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by brutus1213 on Wednesday June 04, 2025 @07:45AM (#44179642)
Amazon is the same I think? I live in constant fear we will have a runaway job one day. I get daily emails to myself (as a manager) and to my finance person. We had one instance where a team member forgot to turn off a machine for a few months :(

I get why it is a business strategy to not have limits .. but I wonder if providers would get more usage if people had more trusts on costs/predictability.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by anonymousab on Wednesday June 04, 2025 @08:16AM (#44179910)
I remember going out to dinner, years ago, with a fairly senior AWS billing engineer. An acquaintance of a coworker.

He looked completely surprised when I asked about runaway billing and why there wasn't any simple options to cap a given resource to prevent those cases.

His response was that they didn't build that because none of their customers wanted anything like that, as far as he was aware.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by mwest217 on Wednesday June 04, 2025 @08:43AM (#44180136)
Disclaimer: I work at Google but not on cloud. Opinions my own.

I think the reason this doesn’t get prioritized is that large customers don’t actually want a “stop serving if I pass this limit” amount. If there’s a spike in traffic, they probably would rather pay the money to serve it. The customers that would want this feature are small-dollar customers, and from an economic perspective it makes less sense to prioritize this feature, since they’re not spending very much relative to customers who wouldn’t want this feature.

Maybe if there weren’t more feature requests to get prioritized this might happen, but the reality is that there are always more feature requests than time to implement them, and a feature request used almost exclusively by the smallest dollar customers will always lose to a feature for big-dollar customers.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by montebicyclelo on Wednesday June 04, 2025 @09:23AM (#44180474)
I guess where it could potentially bring value is by:

Removing a major concern that prevents individuals / small customers from using GCP in the first place; so more of them do use it

That could then lead to value in two ways:

- They make small projects that go on to be large projects later, (e.g. a small app that grows / becomes successful, becomes a moneymaker)

- Or, they might then be more inclined to get their big corp to use GCP later on, if they've already been using it as an individual

But that's long term, and hard to measure / put a number on

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by coredog64 on Wednesday June 04, 2025 @10:01AM (#44180846)
As noted above, there is enough value here such that AWS implemented this several years ago. Said implementation is appropriate for both personal AWS accounts and large scale multi-account organizations.

Having implemented this on behalf of others several times, I'll share the common pain points: * There's a long lead time. You need to enable Cost Explorer (24-48 hours). If you're trying for fine distinctions, activating tags as cost allocation tags is another 24 hours * AWS cost data is a lagging indicator, so you need to be able to absorb a day of charges * Automation support is poor, especially for organizations * Organization budgets configured at the account level are misleading if you don't understand how they're configured

What's really wanted here is that AWS needs to commit to more timely cost data delivery such that you can create an hourly budget with an associated action.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by happyopossum on Wednesday June 04, 2025 @03:22PM (#44184495)
> Said implementation is appropriate for both personal AWS accounts and large scale multi-account organizations.

Followed by a list of caveats that make it wholly irrelevant for an individual who is afraid of a surprise charge covering less than several days.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by jiggawatts on Wednesday June 04, 2025 @10:28PM (#44187796)
Every large enterprise has insurmountable difficult even imagining why customers would want something as bizarre as a "stop loss" on their spending...

... right up until it's their own bottom line that is at risk, and then like magic spending limits become a critical feature.

For example, Azure has no stop-loss feature for paid customers, but it does for the "free" Visual Studio subscriber credits. Because if some random dev with a VS subscription blows through $100K of GPU time due to a missing spending constraint, that's Microsoft's problem, not their own.

It's as simple as that.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by dragandj on Wednesday June 04, 2025 @08:33AM (#44180062)
Yeah, right. Capping a resource, such a wild idea. Of course they won't implement it for the same reason bar owners don't put a cap on drinks.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by ElevenLathe on Wednesday June 04, 2025 @11:12AM (#44181557)
Aren't bars actually required to cap drinks? It's usually phrased as having to refuse serving if you're visibly drunk, but still effectively a cap. That said, a big cloud bill doesn't make you intoxicated. The more I examine this analogy, the less it makes sense.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by mtrovo on Wednesday June 04, 2025 @08:51AM (#44180195)
I don't know if the analogy works that well, the assumption is that you're making more money then you put in the more traffic you get. As a bar owner is the choice between closing your bar for the month when you run out of beer or running to the supplier to bring more kegs.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by 152132124 on Wednesday June 04, 2025 @09:14AM (#44180406)
None of their Big Customers they meant, the small ones who worry about this doesn't matter.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by sidibe on Wednesday June 04, 2025 @08:23AM (#44179975)
I'm sure lot of people at Amazon and Google are aware small customers want this and it's a feature they'd like to brag about, but it is much harder to implement a real time quota on spend than a daily batched job for the money part + realtime resource scoped quotas.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by tmoertel on Wednesday June 04, 2025 @08:47AM (#44180170)
> I get why it is a business strategy to not have limits...

What is the strategy? Is is purely market segmentation? (As in: "If you need to worry about spending too much, you're not the big-money kind of enterprise customer we want"?)

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by nprateem on Wednesday June 04, 2025 @09:24AM (#44180480)
It's not a strategy. It's technically difficult, opens them to liability if runaway happens so fast their system can't stop it, and is only wanted by bottom of the barrel customers.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by physix on Wednesday June 04, 2025 @12:46PM (#44182701)
Just a thought: Maybe if they had some kind of opt-in insurance against overuse until the circuit breaker can kick in?

But, looking from the outside, the lack of protection is effectively a win for them. They don't need to invest in building that out, and their revenue is increased by not having it (if you ignore the effect of throttling adoption). So I have always assumed that there is simply no business case for that, so why bother?

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by coredog64 on Wednesday June 04, 2025 @09:11AM (#44180384)
There's a coarse option: Set up a budget and then a budget action. While ECS doesn't have GPU capabilities, the equivalent here would be "IAM action of budget sets deny on expensive service IAM action" (SCP is also available, but that requires an AWS Org, at which point you've probably got a team that already knows this)

It's coarse because it's daily and not hourly. However, you could also self-service do some of this with CloudWatch metrics to map to a cost and then have an alarm action.

https://aws.amazon.com/blogs/mt/manage-cost-overruns-part-1/

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by delfinom on Wednesday June 04, 2025 @08:43AM (#44180132)
Flaw? Nah

Feature for Google's profits.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by gabe_monroy on Wednesday June 04, 2025 @11:16AM (#44181628)
Heard on this feedback. While not quite a hard cap, I'd also point to https://cloud.google.com/billing/docs/how-to/budgets which many customers are having success with for this use case.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by yarri on Wednesday June 04, 2025 @11:21AM (#44181694)
[edit - Gabe responded]. See this Cloud Run spending cap recommendation [0] to disable billing, which potentially irreversibly deletes resources but does cap spend!

[0] https://cloud.google.com/billing/docs/how-to/disable-billing...

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by badrequest on Wednesday June 04, 2025 @02:38PM (#44184003)
Sure, but why post a tutorial of how to spin this up in GCP instead of...productizing it in GCP?
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by advisedwang on Wednesday June 04, 2025 @05:55PM (#44185969)
It's rock and a hard place for the cloud providers.

Cap billing, and you have created an outage waiting to happen, one that will be triggered if they ever have sudden success growth.

Don't cap billing, and you have created a bankruptcy waiting to happen.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by kamranjon on Wednesday June 04, 2025 @06:16AM (#44179148)
I dunno, the scale to zero and pay per second features seemed super useful to me after forgetting to shut down some training instances with AWS. Also the fast startup ability, if it actually works as well as they say, would be amazing for a lot of the type of workloads that I have.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by isoprophlex on Wednesday June 04, 2025 @06:36AM (#44179256)
Agreed, but runpod or modal offer the same. Happy to use big cloud for a client if they pay the bills, but for personal quests... too scary.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by decimalenough on Wednesday June 04, 2025 @07:29AM (#44179523)
You can set max instances in Cloud Run, which is an effective limit on how much you'll spend.

Also, hard dollar caps are rarely if ever the right choice. App Engine used to have these, and the practical effect was that your website would completely stop working exactly when you least want it to (posted on HN etc).

It's better to set billing alerts and make the call yourself if they go off.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by rustc on Wednesday June 04, 2025 @07:35AM (#44179571)
> Also, hard dollar caps are rarely if ever the right choice.

Depends on if you're a big business or an individual. There is absolutely no reason I would ever pay $100k for a traffic burst on my personal site or side project (like the $100k Netlify case a few months ago).

> It's better to set billing alerts and make the call yourself if they go off.

Billing alerts are not instant and neither is anyone online 24x7 monitoring the alerts.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by brutus1213 on Wednesday June 04, 2025 @07:46AM (#44179653)
100% agreed. This can be solved with technology .. let users set a soft and hard threshold for example. Runaway costs is the problem here.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by ipaddr on Wednesday June 04, 2025 @08:14AM (#44179891)
One bad actor / misconfiguration / attack can put you out of business. It not the safest strategy to allow unlimited liability in business or for personal projects.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by petesergeant on Wednesday June 04, 2025 @07:30AM (#44179534)
I've abandoned DataDog in production for just this reason. Is the amount of money they make on dinging people who screw up really worth the ill-will and people who decide they're just not going to start projects on these platforms?
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by geodel on Wednesday June 04, 2025 @07:19PM (#44186633)
> Is the amount of money they make on dinging people who screw up really worth the ill-will

I think it is .

1) They make money for services they provided instead of looking into meaning of what customer actually wanted.

2) Small time customers move away so they concentrate energy on big enterprise sales.

Not justifying anything here but it just kind of make business sense for them.

Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by petesergeant on Wednesday June 04, 2025 @10:06PM (#44187651)
Definitely possible. I wonder over what time period you miss out on small customers who become big customers and go on that journey with you; perhaps that would be minimal anyway.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by spacecadet on Wednesday June 04, 2025 @07:40AM (#44179605)
Runpod is pretty great. I wrote some genetic end point script that I can deploy in seconds, download the models to the pod, and Im ready to go. Plus I forgot and left a pod running, but down, for a week and it was like 0.60, and they emailed me like 3 times reminding me of the pod.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by nprateem on Wednesday June 04, 2025 @07:21AM (#44179469)
Cloud Run is great but no billing limits is too scary. No idea why they don't address this. They must know if they support individuals we'll eventually leave our saases there.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by randlet on Wednesday June 04, 2025 @08:33AM (#44180055)
Setting max instances effectively caps your spend right?
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by vrosas on Wednesday June 04, 2025 @12:28PM (#44182487)
Yes. CR has had this feature since day 1, people just don't bother to read the docs and would rather write long blog posts blaming their cloud provider for manufacturing the gun they shot themself in the foot with.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by nprateem on Saturday June 07, 2025 @02:04AM (#44207735)
CR is just one service. What if a bug in my app lets users rinse my Gemini API usage, etc.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by rikafurude21 on Wednesday June 04, 2025 @06:30AM (#44179217)
thats what billing limits are for
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by isoprophlex on Wednesday June 04, 2025 @06:35AM (#44179248)
Unless something changed gcp only does billing alerts, not billing limits
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by aiiizzz on Wednesday June 04, 2025 @06:35AM (#44179247)
Those, on gcp, are just alerts, not hard limits, no?
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by jsheard on Wednesday June 04, 2025 @07:01AM (#44179379)
Yeah. I think you can hack together a function which pulls the plug automatically if a billing alert fires, but IIRC the alerts can take a few hours to respond, so extreme runaway usage could still result in a bad time.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by worldsayshi on Wednesday June 04, 2025 @07:21AM (#44179468)
I would rather not realize i had left a bug in that hack after the fact.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by weinzierl on Wednesday June 04, 2025 @09:04AM (#44180325)
I never used modal or vast.ai and from their pages it was not obvious how they solve the yolo billing issue? Are they pre-paid or do they support caps?
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by sharifhsn on Wednesday June 04, 2025 @10:57AM (#44181385)
I know vast.ai uses a prepaid credits system.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by geodel on Wednesday June 04, 2025 @07:20PM (#44186647)
Doesn't seem vast. Seems tight-budget.ai to me :-)
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by thundergolfer on Wednesday June 04, 2025 @12:25PM (#44182468)
Engineer from Modal here: we support caps. They kick in within ~2s if your usage exceeds the configured limit.
Re:Cloud Run GPUs, now GA, makes running AI workloads easier... by oldandboring on Wednesday June 04, 2025 @02:14PM (#44183785)
> uncapped yolo billing

This made me laugh out loud, thank you for this!