News for hackers. Stuff that matters.
Re:Powering cost-efficient AI inference at scale with Cloud ... by londons_explore on Tuesday November 28, 2023 @05:45PM (#38452558)
AI is a big deal this year. Google is one of the biggest tech companies.

Yet a post about AI at Google gets... No comments at all.

Back in 2005, this would have been the most talked about news of the day. Now, nobody cares what Googles up to. They lost their way.

Re:Powering cost-efficient AI inference at scale with Cloud ... by siquick on Tuesday November 28, 2023 @05:56PM (#38452696)
At this point I can’t see why anyone would build their business on any Google product aside from Google Docs.
Re:Powering cost-efficient AI inference at scale with Cloud ... by jeffbee on Tuesday November 28, 2023 @07:42PM (#38453865)
They lost their way or HN readers are out of touch?
Re:Powering cost-efficient AI inference at scale with Cloud ... by krasin on Tuesday November 28, 2023 @08:33PM (#38454291)
Google Drive losing customer data is which one?

Context: https://www.theregister.com/2023/11/27/google_drive_files_di...

Re:Powering cost-efficient AI inference at scale with Cloud ... by jeffbee on Tuesday November 28, 2023 @08:43PM (#38454367)
Actually if you think that Register report supports your first sentence you are perfectly demonstrating that lack of critical thinking that permeates this site.
Re:Powering cost-efficient AI inference at scale with Cloud ... by krasin on Tuesday November 28, 2023 @09:32PM (#38454729)
How do you see the linked issue?
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Tuesday November 28, 2023 @06:12PM (#38452910)
Google, structurally, sucks at anything but sucking down Ad money.

A good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time. As cool as LLM capabilities are the thing that delights me most about them is I can get high quality (YMMV) answers from the internet without all the godforsaken ad bullshit and wonky websites made only as mazes to keep users in ad gardens.

Google is stuck in one hell of an innovators dilemma, one that I can't really see them getting unstuck from.

As far as this particular product, it's touting cost efficiency, but that's not really dominating many decisions yet. There is an all-in sprint to get models trained and take their piece of the pie before the moats get too deep and AI hardware everywhere is back-ordered 12+ months.

Re:Powering cost-efficient AI inference at scale with Cloud ... by oezi on Tuesday November 28, 2023 @06:32PM (#38453152)
> good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time

Until they advertise during your Chat with the LLM. The question is just when this happens not if.

Re:Powering cost-efficient AI inference at scale with Cloud ... by fatherzine on Tuesday November 28, 2023 @07:22PM (#38453664)
If only. How about product placement seamlessly melded into any digital interaction? BRB, prompting ChatGPT for a YCombinator funding pitch.
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Tuesday November 28, 2023 @07:10PM (#38453546)
Too true.

But in the beginning at least, there will be high competition which will hold down the ads a bit, with enshitifcation happening after the market entrenches around the winners.

Re:Powering cost-efficient AI inference at scale with Cloud ... by NBJack on Tuesday November 28, 2023 @09:40PM (#38454788)
LLMs are great at historical data. They however still suck at being kept up to date. I would expect that someone (if not already has) will find a cost efficient way to continuously index new data, but it would be interesting to see how it stacks up with the computing power needed for modern search index pipelines.
Re:Powering cost-efficient AI inference at scale with Cloud ... by deeviant on Wednesday November 29, 2023 @11:09AM (#38461204)
Yeah, when I said "many use cases", that mostly means, "use cases that don't require near-term knowledge." Which are the bulk of my internet searches.

But even know, ChatGPT 4 can use search to get update-to-date info and use it's knowledge store to augment the current information.

Re:Powering cost-efficient AI inference at scale with Cloud ... by humble_nimble on Thursday November 30, 2023 @12:35PM (#38476325)
"As far as this particular product, it's touting cost efficiency, but that's not really dominating many decisions yet"

Why do these LLM startups have such large investments and such large spending? It's because the infrastructure costs of a GenAI startup is huge. That's most of the money that's changing hands. So, from the VCs down to accelerator platforms cost efficiency is a huge decision factor on all of this.

Re:Powering cost-efficient AI inference at scale with Cloud ... by gumballindie on Tuesday November 28, 2023 @07:12PM (#38453569)
Google is no longer a tech heavy company. The only “interesting” things coming out of there are high level executives assigning engineers on random projects against their will, return to office orders, sarcastic offers to relocate workers into company owned hotels so they can “sleep longer” in the morning, and many other mediocre “achievements” of mba type “leaders”.

Just by looking at their core hook, search, you notice a clear reduction in quality. Their core products, ads, are also rife with dubious practices - even more so than in the past. I doubt these are due to technical issues but rather a result of poor product management. Lesson is that no technical company should be led by non technical people. They all turn to shit when that happens.

Re:Powering cost-efficient AI inference at scale with Cloud ... by jauntywundrkind on Tuesday November 28, 2023 @09:27PM (#38454693)
My vague read from before was that TPU4 has been out for a long long time & was basically near impossible to get time in unless you could bid super high.

Not my field & this could be super wrong. But my skepticism that a TPU5e launch was going to help anyone but some niche well connected other big players has been very high. I'm interested from an academic sense, and I love work like the Keras 3.0 release that makes me much more universal/portable, such that hardware like this or others can burn f-ing metal. But I don't have much trust this will actually be available at a reasonable price/with any kind of availability. https://news.ycombinator.com/item?id=38446353 https://keras.io/keras_3/

I'd love to be wrong & have this be actually available at a good price to change what is possible for people, & still interesting. Your call out is right. There's no lightning rod here, nothing to be hyoer-critical of, and few step outside the comfort zone of CUDA have anywhere near enough idea how to assess. This is an unknown on the radar for most.

Re:Powering cost-efficient AI inference at scale with Cloud ... by VirusNewbie on Tuesday November 28, 2023 @10:40PM (#38455246)
My take as a GCP employee: Amazon has really dominated the startup world, but google cloud seems to really be hitting its stride with big enterprise tech companies.

Between its support for K8s, networking stack/infra, and TPUs/ML support we have a lot of huge well known tech companies knocking down our door.

But none of those things really matter as a differentiator for the average startup, scale up, or small business. So all in all, i’m not surprised the majority of HN engineers just aren’t excited by GCP.

I have no knowledge of Cloud’s long term strategy, but it wouldn’t surprise me if by leaning into its success in big tech, they are neglecting the startup scene in terms of marketing or building out new features.