John Carmack trying to get a bunch of GPUs on Google Cloud

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by tveita on Thursday December 01, 2022 @05:41PM (#33824016)

It's a pain. I recently had a compute job that would take a couple of days to run locally. "This is exactly what the cloud is for," I thought and went to Google Cloud where I already had an account.

First try I'm allowed to spec up a beefy server with AVX-512 and click "create", but it never actually starts. Turns out the default CPU limits are tiny - I need to apply to bump two separate limits and wait for that to go through.

Second try I'm actually able to spin up a couple of servers. I set the compute job running and go for lunch. An hour later both are terminated and I get a message about "violation of our Acceptable Use Policy".

There is a link that says crypto mining is forbidden on free instances. I send a reply informing them that a) these are not free instances, and b) I am not mining crypto.

I get a reply back saying more or less "Yes it says free instances but why would you expect anything we write in the documentation to be correct. Also you might think you weren't mining crypto but actually your VM was probably compromised because no one would buy compute instances and actually run compute jobs on them. We're not giving you back access to your instances, maybe delete them and create new ones lol."

I actually find it an easier product to get started with than AWS, but man they really really don't want your money. When you are actively trying to prevent your customers from scaling up resources you are failing at being a cloud.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Thursday December 01, 2022 @06:26PM (#33824535)

Imo these tiny are specifically put in bc a lot of people (especially on this site) always complain how they managed to rack a huge bill by accident. You can’t have it both ways

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by richwater on Thursday December 01, 2022 @06:58PM (#33824863)

I hate this excuse. AWS, GCP etc are not designed for children. They are designed for serious, enterprise-level workloads.

People need to do their due diligence.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by ssmw on Thursday December 01, 2022 @07:03PM (#33824912)

You are going to need a serious, enterprise billing account attached to it to have quotas automatically increased.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Thursday December 01, 2022 @08:24PM (#33825562)

They wouldn’t call it clown compute if it was serious

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by pannSun on Thursday December 01, 2022 @07:29PM (#33825145)

You can't have what both ways, specifically? User-facing spending and capacity limits that are hard to accidentally set too high, without also unilaterally and without warning terminating compute nodes for opaque, alleged ToS violations when their compute actually gets used?

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by fdgsdfogijq on Thursday December 01, 2022 @07:34PM (#33825190)

This isnt why. Its because these services cant scale to everyone having unlimited quotas

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Thursday December 01, 2022 @08:23PM (#33825556)

If you have quota that doesn’t prevent stock out at all which cloud support will be happy to tell you when you have tons pf quota but can’t spin up any more VMs in a region

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by kevincox on Saturday December 03, 2022 @06:42AM (#33842739)

No, but it provides some control. They will stock out if everyone suddenly uses their entire quota (they aren't reservations) but by limiting the quotas that they issue they can reduce the impact of single actors or large swings of smaller actors on availability.

Basically it improves the predictability of resource usages by limiting outlier spikes.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by modeless on Thursday December 01, 2022 @07:51PM (#33825315)

Well maybe if they implemented reasonable ways to cap your bill then they wouldn't need have to have all these arbitrary hidden limits. It's ridiculously difficult bordering on impossible to make sure that a usage spike or bug won't cause an infinitely large bill, on any service they offer. App Engine used to be the exception with an actual spending limit, but (of course) they deprecated and removed it.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Thursday December 01, 2022 @08:15PM (#33825502)

The latency to query cost is on the order of hours on user side so the service would be off the limit by the cost rate * cost latency. That’s my guess for why it’s not a feature.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by 0cf8612b2e1e on Thursday December 01, 2022 @08:55PM (#33825731)

I doubt there is a fundamental limit which says the reporting needs to be that slow. I suspect that some metrics are slower to query than others, but it would go a long way if you could get near instantaneous billing wherever possible. Even better if you could have a hard shut off if cpu/disk/network spend goes over x limit. I choose to believe: it is easy money if people cannot easily avoid accidental mega bills.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Thursday December 01, 2022 @11:15PM (#33826699)

See my other reply. The reason I believe it’s architected this way is to have a good tradeoff between timeliness and overhead cost. Full correctness at global scale would involve serializing all billable events through a series of global transactions, which would bring performance to a crawl (e.g., 10ms just to synchronize the metric).

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by Terretta on Friday December 02, 2022 @12:07PM (#33833267)

This is absolutely right, and I among others have asked AWS to especially please not put any checks on performance "inline" just to do capacity limits. We are happy with the present tradeoffs.

The needs of those wanting to compute should absolutely outweigh the needs of those wanting to not compute.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by modeless on Thursday December 01, 2022 @09:08PM (#33825820)

There is no fundamental reason why they can't do better here. There's no law of physics preventing accurate billing caps. And to the extent that they implement their own caps imperfectly the sensible approach would account for the extra cost as simply part of their costs of running the service which should be budgeted for in aggregate and covered by the pricing that they set themselves. It's simply not a priority for them, clearly.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Thursday December 01, 2022 @10:54PM (#33826561)

There actually are some fundamental laws related to this problem. It’s a distributed system, so you can lose availability of the metrics data and be bounded by the CAP theorem. You’d also need to keep the metrics synchronized at some granularity across zones, effectively serializing all billable events at a global scale. For example, if you wrote one byte into storage across the globe, that write would have to be billed, committed, and be globally visible before the next byte could be written and billed. And each communication is bounded by speed of light, etc. You’d effectively be synchronizing all billable events through a database at cloud scale.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by modeless on Thursday December 01, 2022 @11:31PM (#33826807)

Sure, in theory absolute perfection is difficult. In practice the achievable level of accuracy is well within the bounds of reasonable and far, far beyond what is implemented today. You don't need absolute perfection, since as I suggested you can simply allocate some budget to cover overages amortized over your entire customer base. Especially since the vast majority of the time you'll be operating in a regime where you are far away from the billing cap, so that gives you a lot of leeway. There would never be a need to synchronize every byte written to storage.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Thursday December 01, 2022 @11:52PM (#33826940)

I agree you can get something working. However, if 1% of the time, a customer is overcharged $100+ because they assumed the limit was guaranteed, you’d probably reconsider if you want to offer that service at all.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by modeless on Thursday December 01, 2022 @11:56PM (#33826961)

If 1% of the time a customer uses $100 extra, you don't charge that customer $100 extra. You charge them $0 extra because they had a bill cap and it was your job to enforce that cap. Then you raise the price of the service a very small amount so customers pay $1 more on average and that covers everyone's overages (that's what I mean by amortizing across your customer base). Then you go and improve your limit enforcement because you are actually incentivized to do so, unlike in the case where customers are charged for overages that aren't their fault, where you are actually incentivized to cause overages. Customers aren't dumb, they can see your incentives (see the comment by 0cf8612b2e1e in the adjacent thread).

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Friday December 02, 2022 @12:36AM (#33827209)

It’s plausible to do what you’re saying. I still think it’s more trouble than it’s worth. If you somehow managed to engineer everything perfectly and tune it as you say, you’d still have customers who wanted their service to stay online past overcharge. I’d think the predominant business case for such tight guarantees would be small and low budget projects. Additionally, every cloud vendor who doesn’t do this seems to be cheaper, so you’d be priced out (in a commodity market).

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dwild on Friday December 02, 2022 @11:29AM (#33832669)

> you’d still have customers who wanted their service to stay online past overcharge

Then they wouldn't configure overcharge limit, would they?

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Friday December 02, 2022 @03:38PM (#33836557)

I mean, why not? I have alerts on cost in my cloud usage corresponding to orders of magnitude increases in expected cost. If they trigger a few hours faster (more accurately) that’s good. But I don’t see what I would do bar shutting down the service, so my argument is that it’s not really a feature that would make a difference for most use cases. And if your service is steady, you can already do a linear extrapolation from yesterday’s costs to see if you’d go over budget.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by shalmanese on Friday December 02, 2022 @02:46PM (#33835820)

Except they're in an adversarial relationship with malicious users so as soon as they open up any bit of slack, they open up an attack vector for people to steal their compute resources.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by modeless on Friday December 02, 2022 @03:07PM (#33836136)

I maintain that it is possible within the laws of physics to do a good enough job enforcing quotas that losses are not high, with minimal runtime overhead. The problem is that cloud providers are not incentivized to invest in accuracy, in fact the opposite. Anyway, truly malicious repeat offending customers can be dealt with in other ways.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by codedokode on Friday December 02, 2022 @03:29AM (#33828162)

Isn't main reason that when there are no limits people spend more money?

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by wkdneidbwf on Thursday December 01, 2022 @07:53PM (#33825338)

no way. it’s because they can’t scale to unknown demand.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Thursday December 01, 2022 @11:14PM (#33826696)

Having quota does not guarantee you get the resources. Commit is supposed to (but not really)

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by mirker on Thursday December 01, 2022 @08:08PM (#33825452)

The system approves requests by how trustworthy you are. It does this so: 1. You can’t denial of service the cloud or something else 2. Launder money through mining cryptocurrency 3. Take popular instances away from high profit customers

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by iam-TJ on Friday December 02, 2022 @10:33AM (#33831837)

Seems like it'd be simple to deal with that:

Allow people to pre-pay (deposit) an amount of their choice and set an account flag "Do NOT exceed account balance".

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Friday December 02, 2022 @12:31PM (#33833651)

You will have to do transactional billing at cloud scale which isn’t really feasible

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by g42gregory on Thursday December 01, 2022 @11:04PM (#33826634)

I am curious why would they care if you mine crypto or not? If you pay for an instance, why not?

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by schaefer on Thursday December 01, 2022 @11:20PM (#33826727)

If they don’t fight that use case, I suspect they become a lightning rod for stolen credit card transactions.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by 1123581321 on Thursday December 01, 2022 @11:45PM (#33826890)

It's bad for the rest of us customers. a) more effort goes into fraud, more risk of annoying false positives, b) easy to max out resource utilization which shortens hardware lifespan, c) boom and bust industry makes planning harder which means they can't run on as predictably low margins. This all increases costs and makes free tiers harder to offer. Miners are an easy group to say no to because they don't overlap much with the rest of the software business world, so the reputation hit is small.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by dilyevsky on Friday December 02, 2022 @12:09AM (#33827051)

$20 says 100% of crypto mining in cloud = your customer got p0wned

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by ILike42Cookies on Friday December 02, 2022 @02:20AM (#33827813)

Oh, thats an easy $20.

I mined Ethereum in azure during the height of the crypto boom, and looking at my statistics of vms i was able to get i am pretty sure some other people did to.

On 5k$ spend on azure i got round about 7k$ in Etherum when sold the next day, this includes german Sales tax and some loses i made starting out when not accounting for the german sales tax early on.

I think the whole timeline where this worked was about one and a half weeks(Germany, if you played around with exchange rates and sales tax potentially a bit longer), but it worked pretty well.

Azure was also perfectly fine with me doing it.

[ Parent | on HN ]

Re:John Carmack trying to get a bunch of GPUs on Google Cloud by shalmanese on Friday December 02, 2022 @02:48PM (#33835853)

Yeah, and once that brief period of easy arbitrage was over, 100% of all people who are paying $10 to mine $1 worth of crypto are people who are trying to steal the $10.

[ Parent | on HN ]