News for hackers. Stuff that matters.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by shiftpgdn on Wednesday December 06, 2023 @12:10PM (#38546643)
You must not have been spending million per year. A friend's company spends 10 million/year with GCP, which isn't huge, and can have an engineer from any group in a meeting the next day after a high priority issue.

How frequently are you engaging your account reps? You should be able to get the ear of a PM within 48 hours in most cases.

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by Palmik on Wednesday December 06, 2023 @12:20PM (#38546781)
The parent said "my company that spends many millions per year".
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by pclmulqdq on Wednesday December 06, 2023 @01:14PM (#38547508)
They said that they spend millions per year on GPU compute, not that they spend millions per year with GCP. They later said that they did not spend that money on GCP.

That makes sense because GCP doesn't have all that much capacity for scale-out GPU compute (they offer TPUs for that).

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by leetharris on Wednesday December 06, 2023 @04:57PM (#38550100)
We were trying to get TPUs. We work in audio AI (ASR, TTS, translation, etc). When we saw whisper-jax, we wanted to test the viability of the TPU platform. From our napkin math, the cost/performance ratio seemed great as they always mention on these TPU blog posts.

I posted this in another comment, but the default provisioning for V4 and V5 TPUs is ZERO. They don't tell you this anywhere. So when we'd try to allocate V5 TPUs on our GCP account, it would just fail with a generic error and a huge error number that led to nothing in a search.

So I reach out to our GCP rep we had been working with. After about 10 days and 3 follow-up emails/calls that went unanswered, she replies, "you have to fill out this form." I click on it and it's a Google Form. The same type of Google Form you and I can make.

I submit it. To this day I have heard nothing back. I reach out to various executives at GCP we had been talking with. They said, "you have to fill out a form." I tell them we did, and they say, "oh, it's usually pretty fast." I heard this so many times from so many people. It seems that not a single person who actually works in sales or account management knows how this process works.

When I finally got a response, they told me that "my account has no billing account associated with it." I showed them yes we do, they replied that they were trying to provision for the wrong account. It took another couple weeks to get a follow-up response.

Luckily they eventually connected us with one of their partnered consultants who was finally able to help us, but by then we just decided to go back to GPUs on other platforms because it was such a miserable experience and in that time, all of our providers came through with the volume we needed.

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by shiftpgdn on Wednesday December 06, 2023 @12:23PM (#38546831)
I understand that and am calling that claim into question.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by freedomben on Wednesday December 06, 2023 @01:16PM (#38547540)
and the call out turned out to be right. they clarified that they don't spend those millions with Google. So GP was being honest but the skepticism led to an important clarification, because I read it the same way as parent
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by Palmik on Wednesday December 06, 2023 @02:34PM (#38548425)
I see, my bad!
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by cornel_io on Wednesday December 06, 2023 @12:19PM (#38546764)
Yeah, I spend a small fraction of that and have found GCP support through our account rep to be extremely good, at least on par with what AWS provides. Maybe the particular rep makes a big difference?
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by leetharris on Wednesday December 06, 2023 @12:46PM (#38547160)
We were trying to move to their platform because we needed high-scale TPU or GPU compute. We were not existing customers. We currently spend all those millions elsewhere (AWS, LambdaLabs, Coreweave, and many more) and will continue to do so.

At my previous company, we even had an announced partnership with Google and did lots of co-marketing. That doesn't mean much when a GCP engineer changes something in GCP and breaks your production.

I wish it worked out. I liked working in GCP outside of these problems. I really like GKE.

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by WendyTheWillow on Wednesday December 06, 2023 @12:58PM (#38547315)
FWIW, my experience in AWS, as far as "engineer changes something and breaks your production," is not without its own set of issues. Perhaps the AWS version of this story hasn't impacted you, but AWS also constantly tweaks its services, sometimes to the detriment of its users.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by leetharris on Wednesday December 06, 2023 @04:49PM (#38550009)
I totally believe you. It could absolutely be bad luck / good luck.

I think if you search across the internet you find a lot more GCP horror stories than AWS ones. Considering the market ownership of the two platforms, I'd expect the opposite to be true.

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by belter on Wednesday December 06, 2023 @02:07PM (#38548163)
References please.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by WendyTheWillow on Wednesday December 06, 2023 @04:03PM (#38549430)
You want me to provide references for my personal experience?
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by belter on Thursday December 07, 2023 @09:16AM (#38556548)
Not necessarily. I would like examples of where "...AWS also constantly tweaks its services, sometimes to the detriment of its users..." so I can be better informed, and make better decisions within my projects.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by WendyTheWillow on Thursday December 07, 2023 @01:11PM (#38559686)
And you thought posting, “references please” would be the best way to get that info?

Also those are examples, or details, even. Not references.

Re:Enabling next-generation AI workloads: Announcing TPU v5p... by belter on Friday December 08, 2023 @01:47PM (#38572653)
Are you going to mention, describe or post some examples....Or...keep focusing on the way I asked for it?
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by WendyTheWillow on Friday December 08, 2023 @02:50PM (#38573335)
No, because you've been a jerk during this conversation.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by matthewcford on Wednesday December 06, 2023 @03:08PM (#38548837)
I've seen issues with RDS after an update. We were seeing CPU max out; adding more readers didn't help, and in the end, we had to rebuild the entire cluster. This somehow fixed the issue, and CPU use returned to normal levels; we didn't make any code changes to the app.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by edgoode on Wednesday December 06, 2023 @01:13PM (#38547503)
What company is this? Curious if you'd be open to talking about how you manage compute across providers.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by freedomben on Wednesday December 06, 2023 @01:21PM (#38547604)
I'm not GP but have been in this boat. We tried a number of different approaches, and kubernetes was by far the most successful. Terraform to provision the k8s clusters, workloads deployed to k8s. With OpenShift it gets even better, though I left the project before we finished implementation so I can't say how it went in prod. Early tests were very good though. If you're feeling bold you can do a "stretch" cluster which has nodes in different data centers (some on-prem for example, some in the cloud, or all in the cloud but different zones). The latency between the masters and nodes can cause problems though so I wouldn't separate them geographically very far.
Re:Enabling next-generation AI workloads: Announcing TPU v5p... by usmannk on Wednesday December 06, 2023 @01:33PM (#38547776)
I used to work on cloud infra at an org that spent many millions on GCP (I'm certain ~everyone reading this comment would recognize it by name) and was working on adding more zeros to that number. We spent years working on this and still ended up switching to AWS in the end, largely because of support but also GCP's anti-customer business practices in general. We had weekly meetings with our account reps in person at our office but even they weren't able to get things done internally. No fault of their own though!